There is an interesting conversation on Twitter about Google crawling spikes and (1) if these spikes are global or on a per-site basis and (2) if the URL parameter tool going offline caused a global spike in crawling.
If you don't want to read more, the short answer crawl fluctuations and spikes are not normally a global thing but more on a site-by-site basis. Plus, John Mueller of Google said that the URL parameter tool going offline likely would not result in any spikes to crawling.
Ben Cook and Rebecca Lehmann asked on Twitter:
I'm seeing that, *and* URLs for which we've requested temporary removal that are appearing in the index again after only a month. I thought it was supposed to be six months?
— Rebecca Lehmann (@BeckyLehmann) September 9, 2022
John responded that this would likely be unrelated to that tool going offline:
Got it. Thanks for chiming in.
— Ben Cook (@BenjaminCook) September 10, 2022
Seems like the dial currently is set to “if there’s a link pointing to it, we’re indexing it whether we can crawl it or not.”
Adding that these things are more on a site by site "vibe" and not a global thing:
Oh that’s very interesting!
— Ben Cook (@BenjaminCook) September 10, 2022
Am I correct that if you aren’t allowed to crawl it, you can’t determine the usefulness of the content?
(sorry for the swipo, but that's too good to correct :-))
— 🌽〈link href=//johnmu.com rel=canonical alt 〉🌽 (@JohnMu) September 10, 2022
But many SEOs are indeed noticing this spike:
Reports of "indexed, but blocked by robots.txt" spiking are correct. I have several clients that changed nothing and saw that spike recently -> Google: Crawl Spikes Aren't Global, Normally Noticed On A "Per Site Vibe"
— Glenn Gabe (@glenngabe) September 12, 2022
For example, see screenshot below: https://t.co/j4W7BYq4TQ pic.twitter.com/nioCBiH8pu
I figured I'd highlight this conversation, as I found it somewhat interesting myself.
Forum discussion at Twitter.
