Printer Friendly Version
Email this thread to a friend
|
Doorway Pages (In: Google)
Robots crawling (In: Members Lounge)
MSN dropping pages out the index (In: MSN Search Engine)
Using Robots.txt on Your Web Site (In: General Search Engine Optimization)
Featured Web Site Template |
|
There are 0 guests and 1 members in the forums right now.
Reflects user activity within the last 5 minutes
|
|
| Member |
Message |
serenoo
Joined: Dec 12, 2000
# Posts: 204
|
Posted: 2008-Jan-01 20:02
I need to create a robots.txt file that excludes all pages of my website except for home page and link page.
Pages and directories to disallow are a lot so I cannot list them one by one.
I'd like to disallow everything except / and link.php
Is there a way to make that?
|
 |
g1smd
Staff
Joined: Jul 28, 2002
# Posts: 10418
|
Posted: 2008-Jan-01 20:59
You don't specify filenames in robots.txt, you specify the left-match characters of URLs that need to be blocked.
Disallow: /a will block all URLs that start with a slash and the letter a for example.
So 35 lines, one for each letter /a to /k and /m to /z, and one for each digit /0 to /9 should block almost everything.
Stuff beginning with /l is a lot more tricky. You will need to list out /la to /lh and /lj to /lz individually.
The good news is that you only need to list the ones that actually exist on your site.
|
|
Curt
Joined: Eons Ago
# Posts: 3735
|
Posted: 2008-Jan-02 17:50
g1smd said:
Stuff beginning with /l is a lot more tricky. You will need to list out /la to /lh and /lj to /lz individually.
What makes the letter “l” more difficult? Don't understand the logic.
|
|
g1smd
Staff
Joined: Jul 28, 2002
# Posts: 10418
|
Posted: 2008-Jan-02 20:24
If you block /l in robots.txt, then you will have blocked all URLs that start with a slash and the letter l and that will have also blocked the /links.php URL that the OP didn't want to block.
|
|
Curt
Joined: Eons Ago
# Posts: 3735
|
Posted: 2008-Jan-03 04:27
Oh I see, well uh guess your right.
|
|
You are not permitted to post messages in this forum or topic, because of one or more of the following reasons:
- You have not yet logged in, or registered properly as a member
- You are a member, but no longer have posting rights.
- This is a private forum, for which you do not have permissions.
If you are a recent member, it's possible that you simply have not yet confirmed your account. Please
check your email for a message entitled 'JimWorld Forums: Confirm Your Account' and follow the instructions
contained within.
If you cannot find this message, click here to Re-Send it.
|
If you are still experiencing problem, please read the
Login Assistance
Article for some advice on what may be causing your login not to work properly.
|
Switch to Advanced Editor and ...
Create a New Topic
or Reply to this Thread
|
|
Inbox
