Google Shares Crawl Budget Between Organic & Ads

Nov 17, 2020 - 7:51 am 3 by

Google Crawl Budget Ads Organic

Google shares the same crawl budget between GoogleBot, the organic free web crawler, and Google AdsBot, the paid Google crawler. Keep in mind, Google has dozens of crawlers and they all likely share the same crawl budget.

This is one of the rare instances where Google's Search team shares anything with the Google Ads team, I guess?

The reason this came up is because I saw Microsoft also say crawl budget is shared between organic and ads. So I asked John Mueller of Google if this is the same case at Google. John did not clearly say yes or no. What he said on Twitter was "The goal is to prevent crawling from taking down the server, essentially, so you have to count all the requests, regardless of the crawler type."

Here is the Q&A:

Sounds like that is a yes to me?

I mean, I asked John later for a yes or no answer and he gave me the "it's not new" line - so yes, it is shared:

And yes, it is not new, Glenn Gabe covered John saying this back in March:

And Dawn Anderson shared some screen shots of John saying this years ago:

It makes sense, Google does not want its systems, as a whole, impact your site's performance.

I don't think this is some big conspiracy - it simply makes sense that Google share these resources across all its crawlers, which essentially use the same robots.txt and GoogleBot infrastructure. If not for the case of internal Google efficiencies but also for your own site's health?

But it is also a rare case of where Google Search organic works with Google Ads paid.

John also added this:

Do they share the same index? Generally not...

Forum discussion at Twitter.

 

Popular Categories

The Pulse of the search community

Follow

Search Video Recaps

 
Google AI Overviews, Ranking Volatility, Web Filter, Google Ads AI Summaries & More - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: May 17, 2024

May 17, 2024 - 4:00 pm
Search Video Recaps

Search News Buzz Video Recap: Google AI Overviews, Ranking Volatility, Web Filter, Google Ads AI Summaries & More

May 17, 2024 - 8:01 am
Google Search Engine Optimization

Remove Your Content From Google's AI Overviews

May 17, 2024 - 7:51 am
Google Ads

Google Ads AI Summaries Live For Some Advertisers

May 17, 2024 - 7:41 am
Google Maps

Order with Google For Food Delivery Going Away End Of June

May 17, 2024 - 7:31 am
Google Search Engine Optimization

Two New Googlebots: GoogleOther-Image & GoogleOther-Video

May 17, 2024 - 7:21 am
Previous Story: Microsoft: BingBot & Adixbot Share The Same Crawl Budget Per Site