John Mueller from Google said that the new Content Signals robots.txt directive invented by Cloudflare last year has "no effects whatsoever for any crawler or LLM." He added that it "just adds bloat and future maintenance to your robots.txt file." As far as he knows, he said, none of the crawlers or LLMs use the content-signal robots.txt directives.
This was a response that John Mueller from on Reddit in response to the question, "Testing whether Content-Signal headers and llms.txt actually help with Person entity disambiguation."
Here is what John Mueller posted:
I guess a few things ...* Google doesn't use llms.txt or llms-author.txt. I don't know of any other crawler / llm confirming they're using these (other than SEO tools).
* AFAIK none of the crawlers / llms use the "content-signal" robots.txt directives. It was made up by a CDN, afaik it has no effects whatsoever for any crawler or llm. Using it just adds bloat & future maintenance to your robots.txt file. You can also add other arbitrary things to your robots.txt file, crawlers just use the directives that they support and ignore the rest.
As you may have heard, Cloudflare set a deadline of September 15, 2026 where it will be setting new defaults for each of these three classifications. For all new domains onboarding to Cloudflare, the categories of Training and Agent will be blocked by default on the pages that display ads, while Search will remain allowed by default.
Cloudflare is used by approximately 21.3% of all websites on the Internet as of January 2026. So if you use Cloudflare, you should make sure you want this to happen. I already checked my settings in Cloudflare before that deadline.
That being said, John Mueller is clearly saying that Google has not yet respected the Content Signals robots.txt directive and he makes it sound like the search company has no plans on doing so.
Forum discussion at Reddit.
