Google-Extended Robots Directive Does Not Work For Search Generative Experience

Oct 10, 2023 - 7:41 am 6 by

Robot Singer Google Logo

A couple of weeks ago Google released a new robots.txt directive to tell Google not to use your content for Bard or other AI Google projects - Google-Extended. The Google Search Generative Experience does not currently use Google-Extended, Google told me. That means SGE's AI-generated answers can and will continue to show unless you block Googlebot fully.

Google wrote, that this Google-Extended robots would tell Google not to use your content to improve "Bard and Vertex AI generative APIs, including future generations of models that power those products." Initially I assumed it applied to the AI-generated snapshots provided by Google SGE but it does not.

A Google spokesperson told me, "SGE is a Search experiment so website administrators should continue to use the Googlebot user agent through robots.txt and the NOINDEX meta tag to manage their content in search results, including experiments like Search Generative Experience."

For example, here is an AI-generated answer from SGE that including a card from The Rolling Stones website:

Google Sge Extended Show Example

If you look at their robots.txt file, Google-Extended it listed there:

Rollingstones Google Extended Robots

Glenn Gabe shared another example but by the time I wrote this piece, SGE is no longer showing VentureBeat in SGE for this query:

Since SGE is built into search, Google seems to believe that web publishers are okay with not applying Google-Extended for the AI-snapshots in SGE. "The context is that AI is built into Search, not bolted on, and integral to how Search functions, which is why robots.txt is the control to give web publishers the option to manage access to how their sites are crawled. As you know, we’ve used AI and Large Language Models in Search for many years to not only drastically improve the quality of our results, but also introduce unique ways to search, like Lens and multisearch. These efforts have continued to enhance our ability to connect people with more relevant web pages and send valuable traffic to the ecosystem," a Google spokesperson added.

Your thoughts?

Forum discussion at X.


Popular Categories

The Pulse of the search community


Search Video Recaps

Google AI Overviews, Ranking Volatility, Web Filter, Google Ads AI Summaries & More - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: May 20, 2024

May 20, 2024 - 4:00 pm
Google Search Engine Optimization

Google Working On Surfacing More Content That Comes From The Heart

May 20, 2024 - 7:51 am
Google Search Engine Optimization

Danny Sullivan: We Speak Directly To Engineers Working On Google Ranking Systems

May 20, 2024 - 7:41 am
Google Search Engine Optimization

Comments To Notes In Google Search?

May 20, 2024 - 7:31 am

Google Says Notes On Search Now Ends Soon

May 20, 2024 - 7:21 am
Bing Search

Bing Lets You Turn Off AI Responses But Google Doesn't Let You Turn Off AI Overviews

May 20, 2024 - 7:11 am
Previous Story: Google Lens Can Trigger Search Generative Experience AI Answers