I need a free tool to spider my main URL and bring back all the page titles and decriptions, for all pages connected to that domain. I want it to spider all of my pages for the website.



Any help?

Try Xenu LinkSleuth.


Look in the "sitemap" section of the report that it produces.

Produce a sitemap (there are tons of free sites that offer sitemaps out there). You can also add a search engine tool to your site which will also spider all of your pages (FreeFind or Jrank).

Produce a sitemap (there are tons of free sites that offer sitemaps out there). You can also add a search engine tool to your site which will also spider all of your pages (FreeFind or Jrank).

Sitemap will help to spider all pages of site.

Google it Like Generate Sitemap Online.

Create the Sitemap.xml file into your root folder.

Create the account in Google.
Login with Google account.
Go to Webmaster Tools in Google under my account.. Then you can able to upload new sitemap.xml file and able to know all the features of Google.
Thanks

Let me know if you have any clarifications.

or better yet, put the link to your sitemap in your robots.txt file and google, yahoo and msn will all use the same one and you don't need to submit it anywhere.