Google Crawl Rate Drops: Google Responds?

May 2, 2008 • 8:00 am | comments (9) by twitter Google+ | Filed Under Google Search Engine Optimization
 

Yesterday I reported that GoogleBot is crawling less pages then they once were, based on a large WebmasterWorld thread. Now, I spotted a response from a Googler at a Google Groups thread with similar complaints.

This time, I decided to login to Google Webmaster Tools and check my crawl stats, which can be found under Tools and then at "Set crawl rate." I noticed the same thing for this site, as well as my other sites. Here is a graph from this site:

Google Crawl Rate Dropping

John, the Google representative, blamed the Google Groups thread specific example on a slow site:

Looking at your graphs, it looks like the number of pages that we're crawling has gone down because it's taking so long for us to get each page. In general, there could be any number of reasons why that could be and I think that it may make sense to diagnose it properly, as the others have suggested. Having a slow site is not only bad for search engine crawlers who can't crawl as much as they would like, it's also something your visitors won't be happy about.

But the issue seems way more wide spread then what is in that thread alone and maybe we really do need a response from Google? My traffic referrals from Google seems to be about the same as it was throughout the crawl patterns. But SEOs and webmasters become concern that slow crawl rates and mean slow traffic from that engine in the future.

Forum discussion at WebmasterWorld and Google Groups.

Previous story: Daily Search Forum Recap: May 1, 2008
 

Comments:

gabs

05/02/2008 03:30 pm

Seeing this big time with one of my site.. Massive drop .. No reason as yet...

Verre

05/04/2008 07:32 pm

I too have noticed a significant drop in the number of pages Google is indexing through our webmaster account. In mid-March the crawl rate just dropped off the table. John's explanation is not satisfactory. We had plenty of pages being indexed before, and now, all of the sudden, our site is slow? I doubt it. There's something larger going on.

No Name

05/09/2008 12:28 pm

I had my crawl rate hit almost bottom at one point between march-april and what webmasters are saying is true. slow crawl rate --> less traffic. I just experienced my traffic dropped almost 50% within last two days.

Matt

06/01/2008 02:30 pm

Noticed the same thing. Rankings stay the same, but new pages which were getting crawled in minutes now don't get crawled for 3 days. In Webmaster Tools, I see crawl rate as the same - but then how come new pages dont get indexed? There was an option of faster indexing, so I have set it to that for now.

Bill Ross

06/26/2008 07:33 pm

This too happened across all sites I manage both big and small. We saw a huge drop on encyclopedia.com and highbeam.com and also on some small business sites. So I would think this is not a isolated thing.

Bill Ross

06/26/2008 07:34 pm

I have also just noticed they have removed the graphs like for crawl rate, page load time. Whats up with that?

Richard

02/17/2009 01:39 am

I had the same thing happen to my site. However, I believe it is a Google penalty as I had duplicate content on my site. Has anyone tried to adjust for this?

Rajesh

06/14/2010 06:15 am

Hi, Google is hiding something. Crawl rate has really gone slow though the site is same with all original content and also having good links even for internal pages of my site. Its bad economy which is affecting everyone including google. Google is not spending in resources needed to crawl sites more often. I think they should start charging for crawling.

califmerchant

12/26/2010 07:26 pm

google is ABSOLUTELY crawling less ... some of my sites were last crawled more than a month ago while BING and even YAHOO have visited in the last week!

blog comments powered by Disqus