The other day, I noticed a thread at Google Webmaster Help where a person was complaining that they were being hit hard by GoogleBot. In short, Google's spider was crawling his site in a very aggressive manner. He said:
After setting a custom crawl rate using webmaster tools (and robots.txt for good measure) GoogleBot's crawl rate slowed to the specified 1 request per (approx.) 60 seconds. However, as of a few hours ago the crawl rate has increased to an inexplicable 1 request every 2 to 3 seconds which is unacceptable - I've had to take the drastic response of adding a site wide disallow until I can get this resolved. Why would the crawl rate increase in speed so and how can I get it to return to the values specified?
Googler, Jonathan Simon, replied and said there was a temporary bug that cause the issue. He apologized and said it was now resolved. Jonathan said:
I took a look into what you've reported and it turns out that there was a temporary issue in the processing pipeline of the crawl rate settings which caused them to not take effect. This issue is now resolved so crawl rate settings should once again behave as expected.
I often give Microsoft a hard time about MSNBot being too aggressive and not listening to webmaster directives. So this time, I felt I point out an issue with GoogleBot.
Forum discussion at Google Webmaster Help.
