It is helpful to have a robots.txt file on your server, even if you don't want to block any robot from any directory. Simply create an empty file in a text editor such as Notepad and save it as robots.txt, then upload it to the server.
Why would you do this? Your stats program may show a high number of "page" errors. However, it is probable nearly all of them result from search engine spiders asking for a robots.txt file before they start spidering. If one doesn't exist, an error message is returned. This has no effect on the spiders, because then they will just spider the site, but it creates an error message in your log file. By creating a blank robots.txt file, the search engine spiders will see there are no instructions in it to stop them spidering all pages and no error message is created in your log file.
Thus, the error messages you will see in your stats, will be true errors.
|
