GoogleBot Stop Crawling Your Site On June 15th?

Jun 16, 2010 • 9:00 am | comments (16) by twitter Google+ | Filed Under Google Search Engine Optimization
 

There are several webmasters complaining in Google Webmaster Help and WebmasterWorld that Google has stopped crawling their site's as of yesterday morning.

Some say the crawling has not stopped completely, but rather dipped to a tiny fraction of the crawl rate they normally see.

Even senior member, Walkman, agreed and said:

I got 50 googlebot visits, down from over 1000 normally

There was some speculation that this was only impacting European web sites, but it seems to be larger.

Another senior member said:

Same here, daily average for the last 7 days was 20,000 gbot hits. Yesterday was 150 total gbot hits. Site is a US based site so I don't think this is a European only event.

Is this related to the Google Caffeine launch? I highly doubt it, being that Google Caffeine has been rolling out slowly and slowly over time, so I doubt it is specific to that.

Maybe it has to do with some sort of malware misclassification issue where Google will stop crawling sites until they are de flagged? Maybe it is something completely different. I am not sure.

Forum discussion at Google Webmaster Help and WebmasterWorld.

Previous story: Reciprocal Links: Do They Still Work?
 

Comments:

manof

06/16/2010 04:59 pm

Hi,I report it first in http://www.webmasterworld.com/google/4152588.htm

No Name

06/16/2010 05:35 pm

I look at my web's log. google bot is normal.

No Name

06/16/2010 08:21 pm

I've noticed a real reduction over the last couple years. However, the past few days, super slow.

Nicole

06/16/2010 08:38 pm

I started noticing a real slowdown the last week of May. There were three days where it didn't show at all. Now, it's just a few times a day, 10 to 12 times less than what it has been the rest of this year!

John Mueller

06/16/2010 08:53 pm

It would be useful to know the URLs of the sites involved -- without knowing the sites involved, there's not much we can really look up here. Feel free to post the details here or in the help forum thread. Thanks!

Hobo

06/16/2010 09:19 pm

yup me too

Nitin Gupta

06/17/2010 06:41 am

Hi, I have also feeling something weird from google like the referrals from Google Organic is down and also the referrals from Imgaes.google.com is also down. Have any other faced same problem after may day update & traffic trend is also going down after may 1st. Thanks

serge esteves

06/17/2010 07:54 am

On a french site which count 60000 unique gbot hits per day, since June 15th and June 16, it was 300 gbots hits only.

Víctor Perez

06/17/2010 09:32 am

The problem seems to be already solved. Gbot hits have returned today.

Franz

06/17/2010 09:44 am

googlebot seems to be back, outage lasted about 2 days - verified back on 12 domains.

Drew

06/17/2010 05:48 pm

It is still not crawling the same for me nor indexing new updates. Am I the only one?

Richard

06/18/2010 10:37 am

We had this happen a couple of days ago here in New Zealand. Dropped from about 800,000 pages a day down to about 50,000. Googlebot came back to normal yesterday, but today it's dropped again. Seems to be across the board as in over 20 sites checked they all had big drops in googlebot activity

gabs

06/18/2010 11:24 am

Yep had a look and seeing the same.. Bot's a indexing down across the board.. Just doing a site: search within 24hrs is return such low results.. Seems like a caffeine come down..

Marco

06/24/2010 06:58 pm

Hi, For us it dropped the 15th from 60K a day to 6K... and still not getting up. Have someone find the reason ? thanks,

Lance

11/21/2010 04:57 pm

I saw this same slow down up until just a few days ago. It seems that Googlebot may be back to normal? Also, during the slow down, I was getting a great deal of hits from the feed bot [rate-limited-proxy-209-85-238-234.google.com] in lieu of Googlebot which made me wonder if Google was placing more emphasis on blogs with RSS-ish feeds than just regular websites. I'm glad I wasn't the only one to have noticed this.

Pcsirk

10/26/2011 08:08 pm

looks like its happening again right now.. 15,000 websites and nowhere do we see the gBot..

blog comments powered by Disqus