Printer Friendly Version
Email this thread to a friend
|
Are directories better than files? (In: Google)
Robots crawling (In: Members Lounge)
Using Robots.txt on Your Web Site (In: General Search Engine Optimization)
Featured Web Site Template |
|
There are 0 guests and 1 members in the forums right now.
Reflects user activity within the last 5 minutes
|
|
| Member |
Message |
skoro
Joined: Feb 01, 2004
# Posts: 4
|
Posted: 2004-Feb-17 20:02
I've been running across a few sites and should have been keeping a list on them. I've done a robots.txt file search and couldn't locate any blocking of a search engine spider. Or, I could be doing it wrong too. I'd notice huge directories for link exchanges with no PR and the home page with a PR of 5 or greater. Doesn't make sense, unless your blocking a search engine crawl. I'd like to know if there is another way of catching this type of activity. Maybe the directories are new, and most likely not.
An example would be:
"abcdcompany.com/robots.txt"
Any other ideas on locating blocking of spiders attempts to index pages? My main focust is locating and identifying those sites that are intentially blocking their reciprocal links pages from being indexed at all and thus appearing like they have little outbound links. And what would a diret page block look like on a robot.txt file? Say someone was blocking spiders from accessing a page that looks like:
"abcdcompany.com/directory/businessresources.html"
Thanks Alex
|
 |
g1smd
Staff
Joined: Jul 28, 2002
# Posts: 10438
|
Posted: 2004-Feb-17 20:58
You cross posted this in multiple forums.
Continued in your other thread.
|
|
You are not permitted to post messages in this forum or topic, because of one or more of the following reasons:
- You have not yet logged in, or registered properly as a member
- You are a member, but no longer have posting rights.
- This is a private forum, for which you do not have permissions.
If you are a recent member, it's possible that you simply have not yet confirmed your account. Please
check your email for a message entitled 'JimWorld Forums: Confirm Your Account' and follow the instructions
contained within.
If you cannot find this message, click here to Re-Send it.
|
If you are still experiencing problem, please read the
Login Assistance
Article for some advice on what may be causing your login not to work properly.
|
Switch to Advanced Editor and ...
Create a New Topic
or Reply to this Thread
|
|
Inbox
