Is OpenAI Crawling LLMs.txt Files? Google Says It Won't.

Jul 24, 2025 - 7:21 am 0 by

Openai Robot Spider Web

Recently, Google said that no AI system is currently using the LLMS.txt file. But maybe some are starting to? OpenAI may be starting to discover and crawl LLMS.txt files on websites. While Google's Gary Illyes said they won't be doing that.

Ray Martinez posted a screenshot of his log files showing how OpenAI is crawling his LLMS.txt file every 15 minutes or so. He posted this screenshot on X and wrote, "Log file analysis shows that OpenAI crawls my LLMs.txt file on a few sites. It's pinging our servers every 15 minutes looking for freshness."

Here is the screenshot:

Openai Crawling Llms Txt Files Log

Have you noticed this across any of your sites?

Meanwhile, Gary Illyes from Google, while on stage at the Google Search Central Live Deep Dive event, said Google doesn't support LLMs.txt and isn't planning to. This was captured by Kenichi Suzuki who posted about this on LinkedIn.

Here is a photo:

Google Event

Anthropic has docs on LLMS.txt, so does ElevenLabs and PineCone - as I am sure some others do too. Here is a good blog post on Are LLMs.txt Files Being Implemented Across the Web?

Forum discussion at X.

 

Popular Categories

The Pulse of the search community

Search Video Recaps

 
- YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Google

Google Web Guide Uses Gemini To Organize Your Web Results

Jul 25, 2025 - 7:31 am
Google

Google Shows AI Overviews For Lyrics Explanations

Jul 25, 2025 - 7:11 am
Other Search Engines

Is ChatGPT Using Google Search Results?

Jul 25, 2025 - 5:21 am
Search Forum Recap

Daily Search Forum Recap: July 24, 2025

Jul 24, 2025 - 10:00 am
Google Search Engine Optimization

Google: Normal SEO Works To Get Into AI Overviews

Jul 24, 2025 - 7:51 am
Google

Google Announces Google Trends API

Jul 24, 2025 - 7:41 am
Previous Story: Google AdSense Removing Significant Skin Exposure Block Control Category