GoogleBot Can Also Crawl Too Much & Be Nasty

Nov 20, 2009 - 8:52 am 0 by

The other day, I noticed a thread at Google Webmaster Help where a person was complaining that they were being hit hard by GoogleBot. In short, Google's spider was crawling his site in a very aggressive manner. He said:

After setting a custom crawl rate using webmaster tools (and robots.txt for good measure) GoogleBot's crawl rate slowed to the specified 1 request per (approx.) 60 seconds. However, as of a few hours ago the crawl rate has increased to an inexplicable 1 request every 2 to 3 seconds which is unacceptable - I've had to take the drastic response of adding a site wide disallow until I can get this resolved. Why would the crawl rate increase in speed so and how can I get it to return to the values specified?

Googler, Jonathan Simon, replied and said there was a temporary bug that cause the issue. He apologized and said it was now resolved. Jonathan said:

I took a look into what you've reported and it turns out that there was a temporary issue in the processing pipeline of the crawl rate settings which caused them to not take effect. This issue is now resolved so crawl rate settings should once again behave as expected.

I often give Microsoft a hard time about MSNBot being too aggressive and not listening to webmaster directives. So this time, I felt I point out an issue with GoogleBot.

Forum discussion at Google Webmaster Help.

 

Popular Categories

The Pulse of the search community

Search Video Recaps

 
- YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search News

The Industry Mourns The Loss Of Jill Whalen - Content SEO Hero & Legend

Jun 22, 2025 - 1:20 pm
Search Forum Recap

Daily Search Forum Recap: June 20, 2025

Jun 20, 2025 - 10:00 am
Search Video Recaps

Search News Buzz Video Recap: Google Search Volatility, AI Mode in Search Console, Audio Overviews and ChatGPT Search Quality

Jun 20, 2025 - 8:01 am
Google Search Engine Optimization

Google: Larger & Longer Issues Can Take Longer To Reverse In Google Search

Jun 20, 2025 - 7:51 am
Google Search Engine Optimization

Google Troubleshooting On Blocking Content From AI Mode & AI Overviews

Jun 20, 2025 - 7:41 am
Google

Google Video Search Tests Speaker Icon With Language Label

Jun 20, 2025 - 7:31 am
Previous Story: Google Now Enforcing 35 Character Display URL Policy