How often do LLMs visit llms.txt?

A recent analysis by Profound indicates that the llms-full.txt file receives significantly more AI traffic compared to llms.txt, with ChatGPT being the primary visitor. This preference is attributed to LLMs' inclination to embed comprehensive content upfront, as opposed to using retrieval-augmented generation, which can be hampered by retrieval latency and inconsistent formatting. The study, which examined traffic from 25 companies over a week, found that llms-full.txt provides a richer, more structured dataset, allowing for more efficient indexing and retrieval by AI models like GPT-4-turbo. This format, containing substantially more content than llms.txt, aligns well with large context windows and offers a more streamlined approach to indexing, reducing fragmentation and enhancing retrieval accuracy. While embedding llms-full.txt might initially be more resource-intensive, it ultimately facilitates faster and more consistent AI responses from cached embeddings, highlighting its growing importance in making documentation AI-ready.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.