More Virtual Promote ... Gazette · Webmaster & SEO Tools · Scumware.com · Free Website Templates

SEF

Search Engine Forums

Helping to make the Web - Since 1998

.
Forums	Index	Active Topics	New Topics	My Topics	Search	My Profile	Register	Inbox	Rules & TOS
.

Forum Index · Search Engine Forums · Optimizing Your Website for the Search Engines · Yahoo! · Yahoo releases a new web page spider

Add to hotlist

Mark This Forum Read

Printer Friendly Version

Email this thread to a friend

Moderator(s): yellowwing

Previous Topic Next Topic

Member	Message
coolslko Joined: Aug 22, 2007 # Posts: 10	Posted: 04/16/2008 04:25 am Hi, Recently yahoo released a new web page spider- Slurp 3.0 This new Yahoo! Slurp 3.0 recognizes the same user-agent and all robots.txt directives for 'Yahoo! Slurp,' though it'll identify itself as Slurp 3.0 in your web logs. People will see some changes because of this new web page spider as mentioned below by Sharad Verma from Yahoo Search. "a) The crawlers will start crawling from a different and much smaller set of IP addresses, but it'll still be from the crawl.yahoo.net domain. Any reverse DNS checks to identify our crawler will continue to work. Please note that if you're using IP-based recognition of our crawlers, you might see a drop in crawl/coverage from Yahoo! We strongly recommend that you move to reverse DNS-based identification of Yahoo! Slurp if you're using any other method to avoid this problem. The current set of IPs will disappear from your web logs in the next several weeks. b) The crawlers will also publish a new user-agent, 'Yahoo! Slurp/3.0.' Existing robots.txt directives for 'Slurp' or 'Yahoo! Slurp' will continue to work, but if you have directives specific to 'Slurp/2.0,' they won't be recognized by the new crawler (though usage of the 'Slurp/2.0' user-agent is very rare on the web, so you won't likely be affected). We recommend specifying the shorter version of: User-agent: Slurp. Check out "How do I prevent my site or certain subdirectories from being crawled?" on our Help page for more details. These changes will affect the main Yahoo! Web Search crawlers. Crawlers that similarly respect the Yahoo! Slurp directive but identify themselves more specifically, such as Yahoo! Slurp China and others, will not be impacted." From- [link] (Copy Shortcut)

Forum Index · Search Engine Forums · Optimizing Your Website for the Search Engines · Yahoo! · Yahoo releases a new web page spider

Who's Online?

Reflects user activity within the last 5 minutes

Previous Topic Next Topic

You are not permitted to post messages in this forum or topic, because of one or more of the following reasons:

You have not yet logged in, or registered properly as a member
You are a member, but no longer have posting rights.
This is a private forum, for which you do not have permissions.

If you are a recent member, it's possible that you simply have not yet confirmed your account. Please check your email for a message entitled 'JimWorld Forums: Confirm Your Account' and follow the instructions contained within.

If you cannot find this message, click here to Re-Send it.

If you are still experiencing problem, please read the Login Assistance Article for some advice on what may be causing your login not to work properly.

Switch to Advanced Editor and ... Create a New Topic or Reply to this Thread

Related Forum Topics

Find & Pruchase a “Not For Sale” Web Site? (In: I Want to Sell My Website)
340 sites +3 million page views per month (In: I Want to Sell My Website)
I would like to sell my web site (In: I Want to Sell My Website)
Page Design and Style (In: Graphic Design - Basic HTML, CSS & Usability)
Online forums with focus on Social/Web 2.0 Topics? (In: Directories & Link Building)