Google Checks 4 Billion Host Names Each Day For Robots.txt

Dec 27, 2023 - 7:41 am 1 by
Filed Under Google

Google Robot Paper Filing

Did you know that Google Search checks about four billion host names each and every day for robots.txt purposes? Gary Illyes said in the December Search Off The Record podcast "we have about four billion host names that we check every single day for robots.txt."

He said this at the 20:31 mark in the video. He said if they check four billion host names daily, then "the number of sites is probably over or very likely over four billion."

I spotted this video Glenn Gabe:

Here is the transcript:

GARY ILLYES: Yeah, and I mean, that's one of the problems that we brought up early on. If we implement something or if we come up or suggest something that could work, that should not put more strain on publishers because if you think about it, if you go through our robots.txt cache, you can see that we have about four billion host names that we check every single day for robots.txt. Now, let's say that all of those have subdirectories, for example. So the number of sites is probably over or very likely over four billion.

JOHN MUELLER: How many of those are in Search Console? I wonder.

GARY ILLYES: John, stop it.

JOHN MUELLER: I'm sorry.

GARY ILLYES: Anyway, so if you have four billion hostnames plus a bunch more in subdirectories, then how do you implement something that will not make them go bankrupt when they want to implement some opt out mechanism?

JOHN MUELLER: It's complicated.

GARY ILLYES: It's complicated. And I know that people are frustrated that we don't have something already. But it's not something to--

MARTIN SPLITT: Be taken lightly, yeah.

GARY ILLYES: Yeah.

Here is the video embed at the start time:

Forum discussion at X.

 

Popular Categories

The Pulse of the search community

Follow

Search Video Recaps

 
Google Core Update Coming, Ranking Volatility, Bye Search Notes, AI Overviews, Ads & More - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: July 19, 2024

Jul 19, 2024 - 10:00 am
Search Video Recaps

Search News Buzz Video Recap: Google Core Update Coming, Ranking Volatility, Bye Search Notes, AI Overviews, Ads & More

Jul 19, 2024 - 8:01 am
Google Search Engine Optimization

Billions Of Google goo.gl URLs Will No Longer Work

Jul 19, 2024 - 7:51 am
Google Search Engine Optimization

Google: ccTLDs & Language Do Help You Rank A Little Better In Local Country Region

Jul 19, 2024 - 7:41 am
Google Search Engine Optimization

Google's On Knowing If Your SEO Team Is Doing Their Job

Jul 19, 2024 - 7:31 am
Google Ads

Google Merchant Center Next Gains Support For Supplemental Feeds

Jul 19, 2024 - 7:21 am
Previous Story: Google Notes Feature Sending Emails With Statistics