Google's John Mueller in a hangout from earlier this week said that one of the myths he hears a lot is around crawling, specifically around how SEOs and webmasters define or understand crawl budget.
He said at the 8:34 mark into the video that "at Google we don't really have this notion of crawl budget the way that people are talking about it externally." He added that sometime before the end of the year, Google will come out with a blog post explaining crawl budget in more detail.
He goes on to explain most sites don't need to worry about it. That only really large sites, that generate infinite number of pages, should consider how their servers can handle the load. But most sites do not need to even think about crawl budget.
Here is what he said:
I have a bunch of these around crawling, ignoring your crawl budget will cause your site to sink.So at Google we don't really have this notion of crawl budget the way that people are talking about it externally.
We hope to have a blog post about that sometime soon, probably later this year to explain a bit more about how we handle crawling. But for the most part if you have a reasonably sized site, you don't have to worry about this. You don't have to like hide internal links, you don't have to like mess with no index on individual pages because this page could be using up crawl budget or mess with canonicals or those kind of things for for any reasonably sized site.
Once you have sites that are several thousand pages large or their dynamic and they generate an infinite number of pages, then obviously you want to make sure that your server can handle the load.
It is funny because yes, we covered two recent questions that seemed obviously off, around crawl budget.
Although, Gary Illyes from Google went into more detail about how crawl budget works with Google.
Forum discussion at YouTube Live.
