Yesterday I reported that GoogleBot is crawling less pages then they once were, based on a large WebmasterWorld thread. Now, I spotted a response from a Googler at a Google Groups thread with similar complaints.
This time, I decided to login to Google Webmaster Tools and check my crawl stats, which can be found under Tools and then at "Set crawl rate." I noticed the same thing for this site, as well as my other sites. Here is a graph from this site:
John, the Google representative, blamed the Google Groups thread specific example on a slow site:
Looking at your graphs, it looks like the number of pages that we're crawling has gone down because it's taking so long for us to get each page. In general, there could be any number of reasons why that could be and I think that it may make sense to diagnose it properly, as the others have suggested. Having a slow site is not only bad for search engine crawlers who can't crawl as much as they would like, it's also something your visitors won't be happy about.
But the issue seems way more wide spread then what is in that thread alone and maybe we really do need a response from Google? My traffic referrals from Google seems to be about the same as it was throughout the crawl patterns. But SEOs and webmasters become concern that slow crawl rates and mean slow traffic from that engine in the future.
Forum discussion at WebmasterWorld and Google Groups.
