Google-Extended Robots Directive Does Not Work For Search Generative Experience

Oct 10, 2023 • 7:41 am | comments (6) by | Filed Under Google Search Engine Optimization

Robot Singer Google Logo

A couple of weeks ago Google released a new robots.txt directive to tell Google not to use your content for Bard or other AI Google projects - Google-Extended. The Google Search Generative Experience does not currently use Google-Extended, Google told me. That means SGE's AI-generated answers can and will continue to show unless you block Googlebot fully.

Google wrote, that this Google-Extended robots would tell Google not to use your content to improve "Bard and Vertex AI generative APIs, including future generations of models that power those products." Initially I assumed it applied to the AI-generated snapshots provided by Google SGE but it does not.

A Google spokesperson told me, "SGE is a Search experiment so website administrators should continue to use the Googlebot user agent through robots.txt and the NOINDEX meta tag to manage their content in search results, including experiments like Search Generative Experience."

For example, here is an AI-generated answer from SGE that including a card from The Rolling Stones website:

Google Sge Extended Show Example

If you look at their robots.txt file, Google-Extended it listed there:

Rollingstones Google Extended Robots

Glenn Gabe shared another example but by the time I wrote this piece, SGE is no longer showing VentureBeat in SGE for this query:

Since SGE is built into search, Google seems to believe that web publishers are okay with not applying Google-Extended for the AI-snapshots in SGE. "The context is that AI is built into Search, not bolted on, and integral to how Search functions, which is why robots.txt is the control to give web publishers the option to manage access to how their sites are crawled. As you know, we’ve used AI and Large Language Models in Search for many years to not only drastically improve the quality of our results, but also introduce unique ways to search, like Lens and multisearch. These efforts have continued to enhance our ability to connect people with more relevant web pages and send valuable traffic to the ecosystem," a Google spokesperson added.

Your thoughts?

Forum discussion at X.

Previous story: Google Lens Can Trigger Search Generative Experience AI Answers
Ninja Banner
blog comments powered by Disqus