Google: Crawl Budget Works Differently Than You Think

Dec 8, 2016 - 8:06 am 10 by

Night Sky Google Bot 1900px

Google's John Mueller in a hangout from earlier this week said that one of the myths he hears a lot is around crawling, specifically around how SEOs and webmasters define or understand crawl budget.

He said at the 8:34 mark into the video that "at Google we don't really have this notion of crawl budget the way that people are talking about it externally." He added that sometime before the end of the year, Google will come out with a blog post explaining crawl budget in more detail.

He goes on to explain most sites don't need to worry about it. That only really large sites, that generate infinite number of pages, should consider how their servers can handle the load. But most sites do not need to even think about crawl budget.

Here is what he said:

I have a bunch of these around crawling, ignoring your crawl budget will cause your site to sink.

So at Google we don't really have this notion of crawl budget the way that people are talking about it externally.

We hope to have a blog post about that sometime soon, probably later this year to explain a bit more about how we handle crawling. But for the most part if you have a reasonably sized site, you don't have to worry about this. You don't have to like hide internal links, you don't have to like mess with no index on individual pages because this page could be using up crawl budget or mess with canonicals or those kind of things for for any reasonably sized site.

Once you have sites that are several thousand pages large or their dynamic and they generate an infinite number of pages, then obviously you want to make sure that your server can handle the load.

It is funny because yes, we covered two recent questions that seemed obviously off, around crawl budget.

Although, Gary Illyes from Google went into more detail about how crawl budget works with Google.

Forum discussion at YouTube Live.


Popular Categories

The Pulse of the search community


Search Video Recaps

Google Volatility, Indexing Bugs, Google Ads Broad Match Default & YouTube SEO Tips - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: July 18, 2024

Jul 18, 2024 - 10:00 am
Google Updates

Google Search Ranking Algorithm Volatility - July 18th

Jul 18, 2024 - 7:51 am

Google Tests AI Overviews In Workspace Accounts In The UK

Jul 18, 2024 - 7:41 am

Google Bolding Query In Search Box When Cursor Out Of Search Box

Jul 18, 2024 - 7:31 am
Google Search Engine Optimization

Google Merchant Center Next Coming To All Merchants

Jul 18, 2024 - 7:21 am
Google Search Engine Optimization

Google's Site Reputation Abuse Update Leads Gaming Affiliate To Reduce Financial Forecast

Jul 18, 2024 - 7:11 am
Previous Story: Search Google By Tweeting Emojis To @google