(14)
Carl Malamud, a well respected technologist, author, and public domain advocate posted on Twitter that Bing's spider "got a timeout" by not playing nice with the IRS database. It seems that BingBot was a bit too aggressive on their site and they decided to block Bing for some time.
Here is the tweet:
Bing got a timeout on the IRS database today. Bing was a bad robot. Bing is sitting in a corner, thinking about what it means to play nice.
— Carl Malamud (@carlmalamud) April 25, 2014This caught the attention of Google's Matt Cutts and it drove Carl to add more details:
@mattcutts my server isn't huge, fine with well-behaved bots, they got all piggish. (we ban your ftp robot, but for different reasons.)
— Carl Malamud (@carlmalamud) April 25, 2014It isn't clear exactly which site Carl decided to block. He referenced the IRS database. Bug clearly he is calling BingBot a not well-behaved bot, at least not on April 25th.
Forum discussion at Twitter.
Comments:
Alexander Hemedinger
04/28/2014 01:05 pm
Seems like a fair punishment.
debra mastaler
04/28/2014 01:16 pm
Wonder how MattCutts discovered Mr. Malamud's tweet. ;)-
CaptainKevin
04/28/2014 01:20 pm
Bing probably found pages regarding the Obamacare rules being used by the IRS and got caught in an endless loop of page after page detailing how the IRS will help to carryout a bureaucratic boondoggle.
Lysis
04/28/2014 02:50 pm
Next up: Microsoft gets a full audit from the IRS because it was "randomly" chosen by the "system."
wertwert
04/28/2014 04:18 pm
BingBot is very aggressive. You have to dial it back in Bing Webmaster Tools. Its a little foolish to just outright block it because that just kills 5-10% of your organic traffic. In the IRS's defense bing should dial back their default crawl rate... "Balls to the Wall" shouldn't be the welcome to Bing behavior.
Yo Mamma
04/28/2014 04:49 pm
Loool Barry - How many articles have you written about Google and its problems with investigations and guilty verdicts? Digital Assistant Cortana feature for Windows Phone is leaps ahead of the Siri. Anywho the google bot is way more invasive than anything else out there. If you have a wordpress site - install Wordfence to see how aggressive google bot really is
Barry Schwartz
04/28/2014 04:55 pm
This is about spiders crawling, purely SEO. Nothing to do with any legal issues. Pretty interesting.
vhs
04/28/2014 06:11 pm
Bing is the worst. I hate them. I got like 10000 visitors from Google and 10 from bing.
incrediblehelp
04/28/2014 08:59 pm
Bingbot by far is the most agressive bot in spidering I have ever seen.
Sandeep Mishra
04/29/2014 06:21 am
This is very interesting and nice post. That is the main thing to know and learn the basics of spiders(Bot). This is a fair justice...
Jawad Latif
04/29/2014 07:22 am
Anything which gets viral, also gets attention of Matt especially when it is something related to search engines.
Ben Guest
04/29/2014 02:05 pm
Wasn't it the Sony PlayStation Network that was found in Google's index? However, that was a lack on Sony's part not aggressiveness on Google's.
here
04/29/2014 04:33 pm
we need to ban matt cutts bot.
bob
04/29/2014 04:34 pm
google not giving visitors. also they are not convert well. From bing visitors much better.