Google Crawl Rate Drops: Google Responds?

Yesterday I reported that GoogleBot is crawling less pages then they once were, based on a large WebmasterWorld thread. Now, I spotted a response from a Googler at a Google Groups thread with similar complaints.

This time, I decided to login to Google Webmaster Tools and check my crawl stats, which can be found under Tools and then at "Set crawl rate." I noticed the same thing for this site, as well as my other sites. Here is a graph from this site:

John, the Google representative, blamed the Google Groups thread specific example on a slow site:

Looking at your graphs, it looks like the number of pages that we're crawling has gone down because it's taking so long for us to get each page. In general, there could be any number of reasons why that could be and I think that it may make sense to diagnose it properly, as the others have suggested. Having a slow site is not only bad for search engine crawlers who can't crawl as much as they would like, it's also something your visitors won't be happy about.

But the issue seems way more wide spread then what is in that thread alone and maybe we really do need a response from Google? My traffic referrals from Google seems to be about the same as it was throughout the crawl patterns. But SEOs and webmasters become concern that slow crawl rates and mean slow traffic from that engine in the future.

Forum discussion at WebmasterWorld and Google Groups.

Like The Story? Vote For It On Yahoo Buzz! Or On Sphinn!

posted rustybrick in Google Optimization at May 2, 2008 8:00 AM Comments (7)

Comments

Seeing this big time with one of my site.. Massive drop ..

No reason as yet...

Posted by gabs at May 2, 2008 11:30

I too have noticed a significant drop in the number of pages Google is indexing through our webmaster account. In mid-March the crawl rate just dropped off the table. John's explanation is not satisfactory. We had plenty of pages being indexed before, and now, all of the sudden, our site is slow? I doubt it. There's something larger going on.

Posted by Verre at May 4, 2008 15:32

I had my crawl rate hit almost bottom at one point between march-april and what webmasters are saying is true. slow crawl rate --> less traffic.

I just experienced my traffic dropped almost 50% within last two days.

Posted by No Name at May 9, 2008 08:28

Noticed the same thing. Rankings stay the same, but new pages which were getting crawled in minutes now don't get crawled for 3 days. In Webmaster Tools, I see crawl rate as the same - but then how come new pages dont get indexed?

There was an option of faster indexing, so I have set it to that for now.

Posted by Matt at June 1, 2008 10:30

This too happened across all sites I manage both big and small. We saw a huge drop on encyclopedia.com and highbeam.com and also on some small business sites. So I would think this is not a isolated thing.

Posted by Bill Ross at June 26, 2008 15:33

I have also just noticed they have removed the graphs like for crawl rate, page load time. Whats up with that?

Posted by Bill Ross at June 26, 2008 15:34

I had the same thing happen to my site. However, I believe it is a Google penalty as I had duplicate content on my site. Has anyone tried to adjust for this?

Posted by Richard at February 16, 2009 20:39

Google Crawl Rate Drops: Google Responds?

Comments

Post a comment (Note: Can Take 120 Seconds For Your Comment To Show Up)

Recent Table Activity

SEM Forum Threads

Sponsored Links