Printer Friendly Version
Email this thread to a friend
|
Robots crawling (In: Members Lounge)
Using Robots.txt on Your Web Site (In: General Search Engine Optimization)
Featured Web Site Template |
|
Reflects user activity within the last 5 minutes
|
|
| Member |
Message |
g1smd
Staff
Joined: Jul 28, 2002
# Posts: 10438
|
Posted: 2008-Jan-10 23:33
I'm looking at a site where the robots.txt file for that site is listed in the SERPs in a site:example.com search and appears in the top ten results. That's primarily because some other site links to the robots.txt file and Google has indexed it as if it were any other normal text file.
I am guessing that if I add Disallow: /robots.txt to the robots.txt file, then that will stop the file content from appearing in the SERPs in the future.
I further surmise that Google will still fetch the file to see what is in it, as far as the rule-processing as to what it can and cannot spider on the site is concerned. I mean to say:
Disallow: /
does not stop Google from accessing the robots.txt file to see what is in it, so I don't see why Disallow: /robots.txt should stop it doing so either.
What do you think?
|
 |
mo007
Joined: Dec 13, 2005
# Posts: 52
|
Posted: 2008-Jan-11 17:22
Very good point g1smd,
In my opinion, Disallow: /robots.txt won't stop Google from accessing the file because by the time it sees the directive "Disallow: /robots.txt" it'll be too late!
However, Google should always access to the file robots.txt because that's why it's there in the first place, isn't it? but it should never include it in the SERPs
|
|
Curt
Joined: Eons Ago
# Posts: 3736
|
Posted: 2008-Jan-27 23:04
Wow, that is an ironic situation.
Have you tried to get that other site to delink that file?
You'd think google would not list robots.txt files within SERPs.
|
|
g1smd
Staff
Joined: Jul 28, 2002
# Posts: 10438
|
Posted: 2008-Jan-27 23:26
The robots.txt file was modified about 10 days ago, and the old version is still shown in the SERPs. I would have expected it to disappear by now.
|
|
You are not permitted to post messages in this forum or topic, because of one or more of the following reasons:
- You have not yet logged in, or registered properly as a member
- You are a member, but no longer have posting rights.
- This is a private forum, for which you do not have permissions.
If you are a recent member, it's possible that you simply have not yet confirmed your account. Please
check your email for a message entitled 'JimWorld Forums: Confirm Your Account' and follow the instructions
contained within.
If you cannot find this message, click here to Re-Send it.
|
If you are still experiencing problem, please read the
Login Assistance
Article for some advice on what may be causing your login not to work properly.
|
Switch to Advanced Editor and ...
Create a New Topic
or Reply to this Thread
|
|
Inbox
