Google: Crawl Rate & Crawl Demand = Crawl Budget

Jan 17, 2017 - 7:52 am 2 by

Google Crawl Budget

Several weeks ago, Google told us crawl budget is not what we think it is and promised to write a blog post explaining what Google defines as crawl budget. Well, Gary Illyes posted on the Google blog What Crawl Budget Means for Googlebot.

In short, "crawl rate and crawl demand together," Google said is how they "define crawl budget as the number of URLs Googlebot can and wants to crawl."

Crawl rate limit is how much GoogleBot allows itself to crawl your pages in order not to overload your server and do harm. It is the "number of simultaneous parallel connections Googlebot may use to crawl the site, as well as the time it has to wait between the fetches." Google won't crawl too much to take down your server and it will respect the limit you set in the Google Search Console.

Crawl demand is the demand for Google to crawl your pages, new and old pages. Demand is determined by popularity of your site and URLs and the staleness of the Google systems attempt to prevent URLs from becoming stale in the index. Even if you have more crawl rate left, if demand doesn't require more crawl, GoogleBot won't crawl.

Google said, these two make up crawl budget. "Taking crawl rate and crawl demand together we define crawl budget as the number of URLs Googlebot can and wants to crawl," Google wrote.

What impacts crawl budget?

  • Faceted navigation and session identifiers
  • On-site duplicate content
  • Soft error pages
  • Hacked pages
  • Infinite spaces and proxies
  • Low quality and spam content

Crawl rate is not a ranking factor, said Google. Google wrote, "an increased crawl rate will not necessarily lead to better positions in Search results. Google uses hundreds of signals to rank the results, and while crawling is necessary for being in the results, it's not a ranking signal."

For more details and a lot more FAQs in the blog post.

Forum discussion at Twitter and Google+.


Popular Categories

The Pulse of the search community


Search Video Recaps

Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: May 24, 2024

May 24, 2024 - 10:00 am
Search Video Recaps

Search News Buzz Video Recap: Google Ranking Volatility, Ads In Google AI Overviews, Sundar Pichai Interview, Heartfelt Helpful Content & More Ad News

May 24, 2024 - 8:01 am
Google Search Engine Optimization

Google: The Site Reputation Abuse Policy Enforcement Not Yet Algorithmic

May 24, 2024 - 7:51 am
Google Search Engine Optimization

Google Search Can Now Index Electronic Publication (EPUB)

May 24, 2024 - 7:41 am

Directory Of Embarrassing Google AI Overviews

May 24, 2024 - 7:31 am
Web Analytics

Google Analytics Real-Time Reports Adds Users In Last 5 Minutes

May 24, 2024 - 7:21 am
Previous Story: Tell Google What Technical SEO Topic They Should Write About