Recently, Google said that no AI system is currently using the LLMS.txt file. But maybe some are starting to? OpenAI may be starting to discover and crawl LLMS.txt files on websites. While Google's Gary Illyes said they won't be doing that.
Ray Martinez posted a screenshot of his log files showing how OpenAI is crawling his LLMS.txt file every 15 minutes or so. He posted this screenshot on X and wrote, "Log file analysis shows that OpenAI crawls my LLMs.txt file on a few sites. It's pinging our servers every 15 minutes looking for freshness."
Here is the screenshot:
Have you noticed this across any of your sites?
Meanwhile, Gary Illyes from Google, while on stage at the Google Search Central Live Deep Dive event, said Google doesn't support LLMs.txt and isn't planning to. This was captured by Kenichi Suzuki who posted about this on LinkedIn.
Here is a photo:
Anthropic has docs on LLMS.txt, so does ElevenLabs and PineCone - as I am sure some others do too. Here is a good blog post on Are LLMs.txt Files Being Implemented Across the Web?
Forum discussion at X.
