OpenAI appears to be crawling LLMS.txt files on various sites approximately every 15 minutes. Evidence collected from log files by Ray Martinez demonstrates this activity. Meanwhile, Google has officially stated that it does not support LLMS.txt and has no plans to do so in the future. Other companies such as Anthropic, ElevenLabs, and PineCone have documentation regarding LLMS.txt. The topic has also prompted discussions in forums about its implementation across the web.
Log file analysis shows that OpenAI crawls my LLMS.txt file on a few sites. It's pinging our servers every 15 minutes looking for freshness.
Google doesn't support LLMs.txt and isn't planning to.
Anthropic has docs on LLMS.txt, so does ElevenLabs and PineCone - as I am sure some others do too.
There is a good blog post on Are LLMs.txt Files Being Implemented Across the Web?
Collection
[
|
...
]