Company
Date Published
Author
Tiffany Chen
Word count
848
Language
English
Hacker News points
None

Summary

A recent analysis by Profound indicates that the llms-full.txt file receives significantly more AI traffic compared to llms.txt, with ChatGPT being the primary visitor. This preference is attributed to LLMs' inclination to embed comprehensive content upfront, as opposed to using retrieval-augmented generation, which can be hampered by retrieval latency and inconsistent formatting. The study, which examined traffic from 25 companies over a week, found that llms-full.txt provides a richer, more structured dataset, allowing for more efficient indexing and retrieval by AI models like GPT-4-turbo. This format, containing substantially more content than llms.txt, aligns well with large context windows and offers a more streamlined approach to indexing, reducing fragmentation and enhancing retrieval accuracy. While embedding llms-full.txt might initially be more resource-intensive, it ultimately facilitates faster and more consistent AI responses from cached embeddings, highlighting its growing importance in making documentation AI-ready.