WordPress System Admin Calls Out Google's Evil Spiders

Jun 23, 2010 • 8:20 am | comments (1) by twitter Google+ | Filed Under Google Search Engine Optimization
 

A Google Webmaster Help thread has one of the system administrators from Automattic, the company behind the most popular blogging platform out there, WordPress, complaining about GoogleBot's recent behavior.

He said recent GoogleBot behavior is similar to that of a DoS attack. He explained that "starting around 6:57 EDT today, one of our user's site started getting hammered by dozens if Google Bots instances concurrently, coming from dozens of legitimate IPs in the 66.249.71.n range."

He verified it was from Google's IP range and is concerned that there is a "bug" with Google's crawlers. He then warns Google that WordPress may need to block Google from crawling the blogs if they don't act fast. He said:

We of course do not want to block the Google bot from crwaling this or any of the public content on any of the WordPress.com sites, but since it does not support any of the crawl throttling directives others do (http://www.google.com/support/forum/p/Webmasters/thread?tid=2caa38cb3d55bfa0&hl=en), I'm at a bit of a loss on how to effectively stop this from happening again without taking some drastic measures I'd like to avoid.

JohnMu of Google replied pretty quickly saying:

Thanks for posting -- I'll pass it on to the team here to review. If this is just for one site, using Webmaster Tools would probably be the best way to solve this problem, but it might be useful for our engineers to take a look to see what caused it in the first place.

We have not heard back from Google if it was a bug or if anything changed. Nor did we hear from WordPress if it has been resolved.

But it is pretty cool that WordPress posted this.

Forum discussion at Google Webmaster Help.

Previous story: Google's Maile Ohye On The MayDay Update
 

Comments:

Eddi Hughes

06/29/2010 02:43 pm

I dealt with a similar issue last week with our servers. We had a higher then normal load average last week. Our team started taking a look at it and spoke with the sys admin at Liquid Web. We implemented a bit of security which caused the load average to fall. Then the silence of the storm. Around 12 PST Friday, we started getting hit hard by Google Bots crawling one file, index.php on a non-WordPress related site. The IT System Admin at the datacenter recognized what was going on and blocked it. Over the phone we jokenly exchanged, "Wtf? Someone put an iframe on their network?" Or could they have been 301 redirecting Google Bots like a DoS Attack?

blog comments powered by Disqus