(6)
There are two threads with complaints from webmasters that Microsoft's search crawler, now named BingBot, is crawling way too heavily.
The threads are at BingForums and WebmasterWorld. Yes, the discussion is often very limited when you are watching over Bing, simply because many webmasters and SEOs devote 99% of their attention to Google.
That being said, some are claiming BingBot is acting out of control. Honestly, if it was really bad, I believe more people would be complaining. But Microsoft does have a history of crawling too fast, cloaking their bots and creating fake referrals.
Did you notice a spike in your BingBot traffic and crawling?
Forum discussion at BingForums and WebmasterWorld.
Comments:
GFerrell
06/28/2011 01:16 pm
Based on the last site I took live, my perception is that it takes Bing forever to get through all the pages of a new site. Google is very fast but Bing is so slow that it's embarrassing to report on it's (lack of) progress to clients who want to know how fast their site is being indexed.
guest
06/28/2011 01:40 pm
I'll take "Search Engines" for $700 Answer, "These websites are suddenly getting trounced by MSNbot." Alex, I'm going to have to go with... "What are websites whose visible pagerank went up last week?" Seriously, I'd like to know if the sites getting slammed by MSNbot had their visible pagerank improve right before it happened. I have some good reasons for wondering that which I can't elaborate on in a public blog comment. I'd also like to know if they saw an increase in bot traffic from Baidu Yandex Yahoo at the same time.
targetseo
08/23/2011 06:25 am
On the basis of past lives I took, my understanding is that Bing always get through all the pages of the new site. Google is very fast, but Bing is so slow that it is embarrassing to report it (lack of) progress for customers who want to know how fast their site is indexed. SEO Company India
Anuj A Roy
11/08/2011 11:32 am
Bingbot is crawling our website (yellow pages) heavily since the last week. Prior to that, it was slurp which took our servers by surprise. We were told by Yahoo that they have finished migrating to Bing and the issue will no longer prevail. However, it seems Bing took over the "crawl rampage" from Yahoo. We are still trying to find out the root cause, experimenting with crawl delay for bingbot, temporarily blocking IPs (which is not so advisable by the way). Desperate measures to keep the site up :)
Alex
12/16/2013 02:26 am
sorry for my picture.. same problem with bingbot still crawling our website www.maleradar.com it hapen everyday... how can we stop that???
Lawrence Sheed
12/20/2013 01:36 pm
I covered disabling their BS in 2012, and its happening again, grr. http://www.computersolutions.cn/blog/2012/05/msn-bing-crawler-spider-madness/ My current solution is to block the badly behaved bots from bing using iptables. They do have other ranges in use, so its not a complete block, just a block on the ones that hammer servers. Sample iptables code below: #Log blocks to syslog /sbin/iptables -A INPUT -m limit --limit 5/min -j LOG --log-prefix "PORT 80 DROP: " --log-level 7 #block those %&*%^$ @ bing #Block 207.46.* iptables -A INPUT -s 65.52.0.0/14 -p tcp --destination-port 80 -j DROP #Block 65.52.* iptables -A INPUT -s 207.46.0.0/14 -p tcp --destination-port 80 -j DROP #Block 157.55.17.* iptables -A INPUT -s 157.55.17.0/24 -p tcp --destination-port 80 -j DROP