The other day I reported that Google Crawls 404s After Other Pages So No Crawl Budget Is Sacrificed. So I asked, somewhat in a joking way "so maybe a spike in 404s mean you have some excellent crawl budget expenditure for your site?"
In any event, it seems others asked the question also where David Iwanow asked, if Google is not trying your 404s every now and then, is that a sign of a crawl issue. John Mueller from Google said no, it is not. Here are the tweets:
@davidiwanow @miguelpau @davidwkelley @googlewmc no
— John ☆.o(≧▽≦)o.☆ (@JohnMu) May 11, 2017
So don't freak out if Google isn't trying your 404s, it isn't a bad thing.
Dawn Anderson then reminded me on Twitter that John addressed this topic a year ago. In a video at around the 27 minute mark he went into this. Then at 30:49 into the video John said:
Yeah, I mean usually we use this more to kind of fill up the extra capacity that we have.We say, well, we've crawled all the important URLs from this website. We know we could crawl 10,000 more URLs. So let's just double check the rest of the stuff on our list to make sure that we're not missing anything.
So it's not that you'd lose anything by us not crawling those URLs or you wouldn't gain anything by us not crawling those URLs. It's essentially just our system saying, well, there's this big backlog over here that we don't really know what to do with, so let's just double check it when we have extra time.
If Google has extra capacity, they will go at it. If not, not.
Forum discussion at Twitter.
