GoogleBot Can Also Crawl Too Much & Be Nasty

Nov 20, 2009 - 8:52 am 0 by

The other day, I noticed a thread at Google Webmaster Help where a person was complaining that they were being hit hard by GoogleBot. In short, Google's spider was crawling his site in a very aggressive manner. He said:

After setting a custom crawl rate using webmaster tools (and robots.txt for good measure) GoogleBot's crawl rate slowed to the specified 1 request per (approx.) 60 seconds. However, as of a few hours ago the crawl rate has increased to an inexplicable 1 request every 2 to 3 seconds which is unacceptable - I've had to take the drastic response of adding a site wide disallow until I can get this resolved. Why would the crawl rate increase in speed so and how can I get it to return to the values specified?

Googler, Jonathan Simon, replied and said there was a temporary bug that cause the issue. He apologized and said it was now resolved. Jonathan said:

I took a look into what you've reported and it turns out that there was a temporary issue in the processing pipeline of the crawl rate settings which caused them to not take effect. This issue is now resolved so crawl rate settings should once again behave as expected.

I often give Microsoft a hard time about MSNBot being too aggressive and not listening to webmaster directives. So this time, I felt I point out an issue with GoogleBot.

Forum discussion at Google Webmaster Help.

 

Popular Categories

The Pulse of the search community

Follow

Search Video Recaps

 
Video Details More Videos Subscribe to Videos

Most Recent Articles

Google Ads

Google Ads Suspended 90% More Advertisers This Year & Removed 5.5 Billion Ads

Mar 28, 2024 - 7:51 am
Google Ads

Google Updates Its Definition Of Top Ads; They May Not Be At The Top

Mar 28, 2024 - 7:41 am
Google Ads

Google Ads Adds Share Ad Preview For Performance Max

Mar 28, 2024 - 7:31 am
Google

Google Can Search For Your Blockchain Wallet Addresses (Bitcoin & More)

Mar 28, 2024 - 7:21 am
Google Maps

New Google Shopping & Maps Search Features

Mar 28, 2024 - 7:11 am
Search Forum Recap

Daily Search Forum Recap: March 27, 2024

Mar 27, 2024 - 4:00 pm
Previous Story: Google Now Enforcing 35 Character Display URL Policy