Google's John Mueller said it may make sense to noindex your LLMs.txt file. This way, the file doesn't get indexed by search engines, and then somehow a user lands on it and is confused.
John wrote on Bluesky, "That said, using noindex for it could make sense, as sites might link to it and it could otherwise become indexed, which would be weird for users."
The question was:
Will Google view LLMs.txt files as duplicate content? It seems stiff necked to do so, given that they know that it isn't, and what it is really for.Should I add a “noindex” header for llms.txt for Googlebot?
John Mueller replied in full:
It would only be duplicate content if the content were the same as a HTML page, which wouldn't make sense (assuming the file itself were useful). That said, using noindex for it could make sense, as sites might link to it and it could otherwise become indexed, which would be weird for users.
John previously said that as far as he knows no AI system currently uses llms.txt.
In any event, a lot of sites are adopting it, will you noindex it?
Forum discussion at Bluesky.
