Google To Work On Complementary Robots.txt Protocols For AI & More

Jul 7, 2023 - 7:31 am 3 by

Google Robot Locked Up

Google announced last night that it is looking to develop a complementary protocol to the 30-year-old robots.txt protocol. This is because of all the new generative AI technologies Google and other companies are releasing.

This announcement comes shortly after the news around Open AI accessing paywalled content for its ChatGPT service. But I know many of you are not surprised that Google and others are exploring alternatives to robots.txt with all this generative AI technology floating around the web.

Nothing is changing today, all Google announced was that in the "coming months" they will hold discussions with the "community" to come up with new ideas for a new solution.

Google wrote, "Today, we’re kicking off a public discussion, inviting members of the web and AI communities to weigh in on approaches to complementary protocols. We’d like a broad range of voices from across web publishers, civil society, academia and more fields from around the world to join the discussion, and we will be convening those interested in participating over the coming months."

Google added that it believes "it's time for the web and AI communities to explore additional machine-readable means for web publisher choice and control for emerging AI and research use cases."

What this all means right now, is, I don't know. But here are some responses to my tweet about it:

Gary Illyes from Google, who worked on this protocol over the years, wrote on LinkedIn, "It's time. Nearly 30 years ago robots.txt was born and it served the internet well all this time. With the emerging AI technologies, we need to complement it with new instructions (rules) that were designed for AI applications specifically."

And John Mueller:

If you want to participate, fill out this form.

Do any of you have any ideas?

Forum discussion at Twitter.


Popular Categories

The Pulse of the search community


Search Video Recaps

Google Core Update Coming, Ranking Volatility, Bye Search Notes, AI Overviews, Ads & More - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: July 19, 2024

Jul 19, 2024 - 10:00 am
Search Video Recaps

Search News Buzz Video Recap: Google Core Update Coming, Ranking Volatility, Bye Search Notes, AI Overviews, Ads & More

Jul 19, 2024 - 8:01 am
Google Search Engine Optimization

Billions Of Google URLs Will No Longer Work

Jul 19, 2024 - 7:51 am
Google Search Engine Optimization

Google: ccTLDs & Language Do Help You Rank A Little Better In Local Country Region

Jul 19, 2024 - 7:41 am
Google Search Engine Optimization

Google's On Knowing If Your SEO Team Is Doing Their Job

Jul 19, 2024 - 7:31 am
Google Ads

Google Merchant Center Next Gains Support For Supplemental Feeds

Jul 19, 2024 - 7:21 am
Previous Story: Bing Tests Search Result Listing Branded Side Labels