More Virtual Promote ... Gazette · Webmaster & SEO Tools · Scumware.com · Free Website Templates

SEF

Search Engine Forums

Helping to make the Web - Since 1998

Virtual Promote Member Spotlight

MagazinesQuick.com Discount Magazines

Save up to 95% off the cover price of your favorite magazines! (seekaroo)

.
Forums	Index	Active Topics	New Topics	My Topics	Search	My Profile	Register	Inbox	Rules & TOS
.

Forum Index · Search Engine Forums · Building, Hosting & Maintaining Your Website · SEO Tools & Software Utilities · how 2 create a robot for 40k page website plz help

Add to hotlist

Mark This Forum Read

Printer Friendly Version

Email this thread to a friend

Moderator(s): Prowler

Previous Topic Next Topic

Member	Message
godwinseo Joined: Sep 24, 2005 # Posts: 2	Posted: 09/24/2005 06:19 am Hi I need 2 create a robot.txt file for a site woth 40000 web pages.how do i do that?is there any tools that you wud recommend?plz help me and its very urgent thank u godwin (Copy Shortcut)

Logan Moderator Joined: Aug 14, 2002 # Posts: 3749 Website	Posted: 09/24/2005 07:49 am for starters its robots.txt Do you want to exclude all 40000 web pages from all of the search engines? If so this can simply be done at the root level. You can also specify exclusion by directory or individual url. It is not necessary to exclude a whole site by individual url, so defining what you actually want to do is necessary. Its just a .txt file so its not a very popular function as you describe for 'automated tools'. (Copy Shortcut)

godwinseo Joined: Sep 24, 2005 # Posts: 2	Posted: 09/24/2005 09:36 am Hi Logan thanks for ur kind interest.the scenario is like this.for small sites i just manually check the pages indexed by google and see if there are any pages that are not working anymore.Then i include such pages in robots.txt file.Now how do i do this for a site which has 40k pages.i certainly cannot afford to do it manually.so wats the way to do this? plz help me out of this. thank u godwin (Copy Shortcut)

g1smd Moderator Joined: Jul 28, 2002 # Posts: 10060 Website	Posted: 09/24/2005 01:28 pm Google only reads the first fifty lines in the robots.txt file. (Copy Shortcut)

Forum Index · Search Engine Forums · Building, Hosting & Maintaining Your Website · SEO Tools & Software Utilities · how 2 create a robot for 40k page website plz help

Who's Online?

Reflects user activity within the last 5 minutes

Previous Topic Next Topic

You are not permitted to post messages in this forum or topic, because of one or more of the following reasons:

You have not yet logged in, or registered properly as a member
You are a member, but no longer have posting rights.
This is a private forum, for which you do not have permissions.

If you are a recent member, it's possible that you simply have not yet confirmed your account. Please check your email for a message entitled 'JimWorld Forums: Confirm Your Account' and follow the instructions contained within.

If you cannot find this message, click here to Re-Send it.

If you are still experiencing problem, please read the Login Assistance Article for some advice on what may be causing your login not to work properly.

Switch to Advanced Editor and ... Create a New Topic or Reply to this Thread

Related Forum Topics

340 sites +3 million page views per month (In: I Want to Sell My Website)
Developed adult videochat website for sale. (In: I Want to Sell My Website)
Greeting Card Website for sell (In: I Want to Sell My Website)
Page Design and Style (In: Graphic Design - Basic HTML, CSS & Usability)
Website Redesign Issue (In: Google)