Google: Crawl Budget Works Differently Than You Think

Dec 8, 2016 - 8:06 am 10 by

Night Sky Google Bot 1900px

Google's John Mueller in a hangout from earlier this week said that one of the myths he hears a lot is around crawling, specifically around how SEOs and webmasters define or understand crawl budget.

He said at the 8:34 mark into the video that "at Google we don't really have this notion of crawl budget the way that people are talking about it externally." He added that sometime before the end of the year, Google will come out with a blog post explaining crawl budget in more detail.

He goes on to explain most sites don't need to worry about it. That only really large sites, that generate infinite number of pages, should consider how their servers can handle the load. But most sites do not need to even think about crawl budget.

Here is what he said:

I have a bunch of these around crawling, ignoring your crawl budget will cause your site to sink.

So at Google we don't really have this notion of crawl budget the way that people are talking about it externally.

We hope to have a blog post about that sometime soon, probably later this year to explain a bit more about how we handle crawling. But for the most part if you have a reasonably sized site, you don't have to worry about this. You don't have to like hide internal links, you don't have to like mess with no index on individual pages because this page could be using up crawl budget or mess with canonicals or those kind of things for for any reasonably sized site.

Once you have sites that are several thousand pages large or their dynamic and they generate an infinite number of pages, then obviously you want to make sure that your server can handle the load.

It is funny because yes, we covered two recent questions that seemed obviously off, around crawl budget.

Although, Gary Illyes from Google went into more detail about how crawl budget works with Google.

Forum discussion at YouTube Live.

 

Popular Categories

The Pulse of the search community

Follow

Search Video Recaps

 
Google Core Update Volatility, Helpful Content Update Gone, Dangerous Search Results & Ads Confusion - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: April 18, 2024

Apr 18, 2024 - 4:00 pm
Google

Google Notes On Search Won't Necessarily Go Away In May

Apr 18, 2024 - 7:51 am
Google Maps

Google Maps Releases New Directions, Travel & EV Features

Apr 18, 2024 - 7:41 am
Google Ads

Google Ads Reminds Advertisers Some Ad Customizers Will Go Away May 31st

Apr 18, 2024 - 7:31 am
Google Search Engine Optimization

Google Drops Video Carousel Markup

Apr 18, 2024 - 7:21 am
Google Maps

Google Business Profiles Register Your Defibrillator (AED)

Apr 18, 2024 - 7:11 am
Previous Story: Search Google By Tweeting Emojis To @google