Printer Friendly Version
Email this thread to a friend
|
Great money with little effort is the key (In: Members Lounge)
Featured Web Site Template |
|
There are 0 guests and 1 members in the forums right now.
Reflects user activity within the last 5 minutes
|
|
| Member |
Message |
StupidScript2
Joined: Mar 23, 2004
# Posts: 183
|
Posted: 2004-May-27 17:39
Yikes!
Looking at my log files, today, and I see MSNBot has stepped up its usually greedy pace. Incredibly, we have had over 717,500 hits and 16.6 GB of data transferred to MSNBot this month alone. The next closest bot is Inktomi/Slurp at 19,640 hits and 488.5 MB of data.
I started checking MSNBot's hits a few days ago when their data transfers hit 4 GB...and that was only a few days ago.
Has anyone else experienced this? I'm thinking about robots.tx-ing MSNBot out of my system for a while, to cool it down, but I don't know if it will drop me completely or what.
My hosting account is good for 5 GB of data transfer per month before I start getting charged. It's clear I will be paying extra, this month...
|
 |
Mike-Levin.com
Joined: Sep 27, 2000
# Posts: 1988
|
Posted: 2004-May-27 21:03
I haven't seen that much of an increase, but MSN is picking up a few hundred pages a day now, as opposed to only a few dozen a day before. It's a 10x increase in MSNBot activity, but it still doesn't match Yahoo or Google's bots.
|
|
StupidScript2
Joined: Mar 23, 2004
# Posts: 183
|
Posted: 2004-May-27 22:30
I've adjusted my robots.txt file to keep MSNBot from spidering me, for the time being. I'll re-adjust next week and see if it's calmed down a bit.
Is MSN using its bot index, or are they just gathering data for a launch?
|
|
StupidScript2
Joined: Mar 23, 2004
# Posts: 183
|
Posted: 2004-May-28 02:22
Oh, man! How could I not see it...picture this:
MS is working to put together a so-called "proprietary" search function for use in their universe of products: web, desktop, etc.
They need an edge. They have an idea.
They send out their crawler to gather as much of their "edge" as it can before the launch of the search function.
Their "edge"? Multimedia found and delivered faster than the rest.
Their trick? Lots and lots of servers hosting their cached content. All it takes is money.
Their content? Lots and lots of videos, audio files, Flash files and presentations.
(We have several videos in .WMV and .RM format on our sites. Not GBs...but if requested repeatedly, it adds up.)
The crawler caches the media files, and the search function doesn't need to wait for connections to the original servers or rely on their pipe, anymore.
Plus, they have total authority to lay whatever kind of ads, etc. they want over the delivered content, as it's their server, and their authority.
Oh, MAN! Time for a splash of cold water. This fantasy is getting out of hand...
|
|
StupidScript2
Joined: Mar 23, 2004
# Posts: 183
|
Posted: 2004-May-28 17:23
The good news? MSNBot is respecting my robots.txt file. No hits frlom them since I updated it to exclude them.
Whew! 
|
|
bhartzer
Staff
Joined: Jun 08, 2000
# Posts: 7042
|
Posted: 2004-May-28 17:32
I keep them out until I see their index and their SERPs.
|
|
StupidScript2
Joined: Mar 23, 2004
# Posts: 183
|
Posted: 2004-Jun-08 18:59
MSNBot FAQ: http://search.msn.com/webmasters/msnbot.aspx
Lots of folks are reporting greedygreedy MSNBot in other forums, similar to my experience, above.
MSNBot is, indeed, in prototype condition at this time. It's building its index, and the results are not currently being included in MSN searches.
|
|
bhartzer
Staff
Joined: Jun 08, 2000
# Posts: 7042
|
Posted: 2004-Jun-08 19:08
I'm seeing more and more that people are keeping MSNbot from indexing their sites. That's what I suggest now until we see their index.
|
|
mteasdal
Joined: Eons Ago
# Posts: 376
|
Posted: 2004-Jun-10 17:56
I have a new site (600 pages)that MSNBOT indexes almost the entire site every other day. I have read this is Microsoft research but I question if other engines are using what it is finding. I added 150 pages yesterday and saw the MSNBOT hit them last night. Everyday now I am using seekscan to watching a dozen search engines to see if they pickup these new pages. I made changes to the title and other info on those 150 pages to help detect this.
I suspect TEOMA is using the msnbot or gigabot. I hope to prove it this week. If I can prove it I will post it.
|
|
Bry
Joined: Sep 30, 2003
# Posts: 207
|
Posted: 2004-Jun-17 23:07
You added 150 pages to your web site in a day? It always boggles my mind to see statements like that.
MSN must not like me. It only came 27 times this month compared to Googlebot's 269. Yahoo is apparently used to me since they only came once this month.
|
|
mteasdal
Joined: Eons Ago
# Posts: 376
|
Posted: 2004-Jul-07 05:32
Bry - I have a custom database I press a button and it writea hundreds of html pages. Giving fresh html pages to the bots to feed on.
OK I can not find any search engine using this msn bot so I did as others.
My robots.txt got a new few lines
User-agent: msnbot
Disallow: /
When I see msn bot doing something for me I will let it back in.
|
|
mteasdal
Joined: Eons Ago
# Posts: 376
|
Posted: 2004-Jul-14 23:28
I found a link to the new MS test site in my logs.
The MSNBOT is collecting data and it can be searched at the MS dev site.
http://techpreview.search.msn.com/
I will allow the msnbot back into my sites because I have read the site may come up in the coming few months. Can anyone confirm this? When it does I wan to have my pages indexed fully.
My new site is not in msn.com but it is indexed well in the new msnbot techpreview site.
|
|
StupidScript2
Joined: Mar 23, 2004
# Posts: 183
|
Posted: 2004-Jul-21 22:51
Well that stinks!
After MSN gobbled up the above-mentioned 16GB+ in bandwidth from our sites, they are NOWHERE to be found in the index the new dev site.
Thanks, Microsoft, for yet another crappy system!
|
|
You are not permitted to post messages in this forum or topic, because of one or more of the following reasons:
- You have not yet logged in, or registered properly as a member
- You are a member, but no longer have posting rights.
- This is a private forum, for which you do not have permissions.
If you are a recent member, it's possible that you simply have not yet confirmed your account. Please
check your email for a message entitled 'JimWorld Forums: Confirm Your Account' and follow the instructions
contained within.
If you cannot find this message, click here to Re-Send it.
|
If you are still experiencing problem, please read the
Login Assistance
Article for some advice on what may be causing your login not to work properly.
|
Switch to Advanced Editor and ...
Create a New Topic
or Reply to this Thread
|
|
Inbox
