Olivier Papon from Seolyzer, a log analysis toolset, reported the other week that Google seemed to slow or stop crawling most of the web between around November 11 and November 17th. I did ask Google about this but have yet to hear back. It seems Googlebot is back at the crawling business now but we are not sure what went wrong or if something did go wrong. Update, Google confirmed the issue - saying it was not widespread - see below for more details.
Olivier posted on Twitter on November 15th about this saying "Googlebot is on strike! Googlebot has drastically reduced its crawl activity on many large sites since November 11 at 6PM (GMT)." Many others said they noticed the same thing. I emailed Google about it and they said they would get back to me but I didn't really hear back.
John Mueller of Google did tweet that "We've been looking into it." But I have not heard anything else from Google, yet.
Then on November 18th, Olivier said "Googlebot is back! Googlebot came back as he left, at a specific time: November 17 at 8PM." He shared this chart showing how Googlebot went away and came back:
I personally went through tons of crawl stats reports in Google Search Console to verify it throughout the week. I'd say 90% or more of those who I checked, showed zero dip in crawl activity. That is with the exception of some but I am not sure if there was a pattern. Here is one sample chart showing the dip in crawl activity.
Some others noticed this as well:
🤖 #SEO - Crawl Rate Fixed 🤖
— Praveen Sharma (@MusingPraveen) November 22, 2021
🔸 It seems like the recent Googlebot crawl rate issue which we faced earlier has been fixed now. :) pic.twitter.com/AXUeEDmzNZ
Again, I have zero confirmation from Google yet on what this issue was, how many sites were impacted and how they may have been impacted.
You can click through to these tweet below to see more details:
🔥Googlebot is back! 😅
— Olivier @Seolyzer.io 🚀 (@Seolyzer_io) November 18, 2021
Googlebot came back as he left, at a specific time: November 17 at 8PM. Look the thread below to see the hole in the chart 🔽 What happened ? https://t.co/VmnDjeoAYW
Forum discussion at Twitter.
Update: Here is John Mueller of Google confirming the issue:
Yes, we saw that crawling for the caches had slowed down for some sites. This bug has since been resolved, and an internal analysis showed it wasn't widespread, and that there were no negative effects overall.
— 🧀 John 🧀 (@JohnMu) November 23, 2021
This has been resolved for a while now, just finally remembered that Barry had asked :). I'm guessing all crawl-graphs will be back at normal in the meantime.
— 🧀 John 🧀 (@JohnMu) November 23, 2021
