DigitalPoint Founder Upset With MSNBot's Crawl Rate (MSNBot 2.0b)

Aug 3, 2009 - 8:19 am 1 by
Filed Under Bing Search

Shawn Hogan, DigitalPoint's founder, has posted a thread at DigitalPoint Forums clearly showing his frustration with MSNBot, Microsoft Bing's search crawler. He is upset that the bot is crawling too much, too fast - causing an unnecessary spike in load on his servers.

Based on his metrics, he noticed MSNBot/2.0b was crawling at a rate of 8Mbit per second, all day, every day.

Microsoft did recently announced that this beta crawler, 2.0b (b is for beta), will be ramping activity and it seems like DigitalPoint was hit by this. You might be also, so make sure to check out your bot activity and as the Microsoft post says:

If you have any feedback for us on either the existing or the new MSNBot, or on any other matters pertaining to crawling your website, please post your comments and questions in our crawling/indexing forum. Our Webmaster Center forums moderator will follow-up with you ASAP. Please include in your forum post the identity of the user agent that is the source of your concern so we can help you that much faster. Thank you!

Microsoft's beta bots have a history of causing trouble for webmasters.

Forum discussion DigitalPoint Forums.

 

Popular Categories

The Pulse of the search community

Follow

Search Video Recaps

 
Google Core Update Volatility, Helpful Content Update Gone, Dangerous Search Results & Ads Confusion - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: April 16, 2024

Apr 16, 2024 - 4:00 pm
Google Search Engine Optimization

Google Will Fight Site Reputation Abuse Spam Both With Manual Actions & Algorithms

Apr 16, 2024 - 7:51 am
Google

Google Search Tests Thumbs Up/Down Buttons In Product Grid Results

Apr 16, 2024 - 7:41 am
Google Ads

Google Ads "Similar Product" Carousel

Apr 16, 2024 - 7:31 am
Google Search Engine Optimization

Clarification: Google Search Supports Images Referenced From src Attribute

Apr 16, 2024 - 7:21 am
Google AdSense

Google AdSense New Ad Intents Formats - Links & Anchors In Content

Apr 16, 2024 - 7:11 am
Previous Story: Google Site Reconsideration Form Not Working For Many