GoogleBot Can Also Crawl Too Much & Be Nasty

Nov 20, 2009 - 8:52 am 0 by

The other day, I noticed a thread at Google Webmaster Help where a person was complaining that they were being hit hard by GoogleBot. In short, Google's spider was crawling his site in a very aggressive manner. He said:

After setting a custom crawl rate using webmaster tools (and robots.txt for good measure) GoogleBot's crawl rate slowed to the specified 1 request per (approx.) 60 seconds. However, as of a few hours ago the crawl rate has increased to an inexplicable 1 request every 2 to 3 seconds which is unacceptable - I've had to take the drastic response of adding a site wide disallow until I can get this resolved. Why would the crawl rate increase in speed so and how can I get it to return to the values specified?

Googler, Jonathan Simon, replied and said there was a temporary bug that cause the issue. He apologized and said it was now resolved. Jonathan said:

I took a look into what you've reported and it turns out that there was a temporary issue in the processing pipeline of the crawl rate settings which caused them to not take effect. This issue is now resolved so crawl rate settings should once again behave as expected.

I often give Microsoft a hard time about MSNBot being too aggressive and not listening to webmaster directives. So this time, I felt I point out an issue with GoogleBot.

Forum discussion at Google Webmaster Help.

 

Popular Categories

The Pulse of the search community

Follow

Search Video Recaps

 
Google Core Update Rumbling, Manual Actions FAQs, Core Web Vitals Updates, AI, Bing, Ads & More - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Google Updates

Google Urges Patience As The March 2024 Core Update Continues To Rollout

Mar 18, 2024 - 7:51 am
Google

Official: Google Replaces Perspective Filter With Forums Filter

Mar 18, 2024 - 7:41 am
Google Maps

Google Business Profiles Now Offers Additional Review After Appeal Is Denied

Mar 18, 2024 - 7:31 am
Google Maps

EU Searchers Complaining About Google Maps Features Changes Related To DMA

Mar 18, 2024 - 7:21 am
Google

Google Showing Fewer Sitelinks Within Search

Mar 18, 2024 - 7:11 am
Search Forum Recap

Daily Search Forum Recap: March 15, 2024

Mar 15, 2024 - 4:00 pm
Previous Story: Google Now Enforcing 35 Character Display URL Policy