Hi
I need 2 create a robot.txt file for a site woth 40000 web pages.how do i do that?is there any tools that you wud recommend?plz help me and its very urgent
thank u
godwin

for starters its robots.txt

Do you want to exclude all 40000 web pages from all of the search engines? If so this can simply be done at the root level. You can also specify exclusion by directory or individual url.

It is not necessary to exclude a whole site by individual url, so defining what you actually want to do is necessary.

Its just a .txt file so its not a very popular function as you describe for 'automated tools'.

Hi Logan
thanks for ur kind interest.the scenario is like this.for small sites i just manually check the pages indexed by google and see if there are any pages that are not working anymore.Then i include such pages in robots.txt file.Now how do i do this for a site which has 40k pages.i certainly cannot afford to do it manually.so wats the way to do this?
plz help me out of this.
thank u
godwin

Google only reads the first fifty lines in the robots.txt file.