SEF
Search Engine Forums
Helping to make the Web - Since 1998
|
Virtual Promote Member Spotlight
How To Make Money with Your Blog
My 200+ page book details how to build, optimize and monetize a blog. (SportsGuy) |
| . | ||||||||||
| Forums | Index | Active Topics | New Topics | My Topics | Search | My Profile | Register | Inbox | Rules & TOS | |
| . | ||||||||||
Email this thread to a friend
| Previous Topic Next Topic |
| Member | Message |
|
moverworldwide Joined: Mar 01, 2006
# Posts: 35
|
Posted: 03/24/2006 10:14 pm
hello, sorry for my ignorance. i have my site up on google and yahoo. recently i checked with google sitemap and google says there is no robots.txt file. i do not know what is this stuff. i read through many forums QA but no where there is a guideline for a beginer. can anyone help me how should i create a robots.txt file? i understood from forums to create one in note pad and name it robots.txt. but what is the stuff that i have to put in for all search engines to crawl all my pages? Do i have to register with any robots company or just crate a robots.txt file and upload to the root of my site?? how can i check whether my robots.txt file created is working?? tnks for any help  |
 | |
|
Prowler Moderator
Joined: Aug 14, 2000
# Posts: 1653
|
Posted: 03/27/2006 06:35 am
Robots.txt is a text file in the root - containing directives to robots. Specifically the directives indicate to the robots which directories/files are disallowed. All mainstream search engine robots obey the directives while many rogue robots pick out hidden directories from this file. You don't need a robots.txt or you may have a blank robots.txt which indicates that the entire site can be crawled at will. The following is a typical robots.txt : Here is one online validator: [link] More detailed information about the robots.txt than we have the space here (for the terminally impatient readers) head to this link: [link] |
| |
|
iann2006 Joined: Apr 02, 2006
# Posts: 5
|
Posted: 04/02/2006 09:16 pm
The robots.txt file is a set of instructions for visiting robots (spiders) that index the content of your web site pages. The file must reside in the root directory of your web. For those spiders that obey the file, it provides a map for what they can, and cannot index. To exclude all robots from the server (do not use this one unless you want no indexing for the entire site!): User-agent: * Disallow: / To exclude all robots from parts of a server: User-agent: * Disallow: /private/ Disallow: /images-saved/ Disallow: /images-working/ To exclude a single robot from the server: User-agent: Named Bot Disallow: / To exclude a single robot from parts of a server: User-agent: Named Bot Disallow: /private/ Disallow: /images-saved/ Disallow: /images-working/ Note: The asterisk (*) or wildcard in the User-agent field is a special value meaning "any robot" and therefore is the only one needed until you fully understand how to set up different User-agents. If you want to Disallow: a particular file within the directory, your Disallow: line might look like this one: Disallow: /private/top-secret-stuff.htm  |
| |
Who's Online?
There are 175 guests and 2 members in the forums right now.
Reflects user activity within the last 5 minutes
| Previous Topic Next Topic |
Switch to Advanced Editor and ... Create a New Topic or Reply to this Thread |
Related Forum Topics
© 1995 - 2006 · iWeb, Inc · DBA JimWorld Productions