llms.txt is still overhyped

A year after initially criticizing llms.txt, the author reports that server logs, an Ahrefs analysis, and statements from Google continue to show little evidence that AI crawlers or chat systems routinely fetch or use the root-level manifest, while most observed requests come from SEO tools, scanners, or site owners themselves. Although adoption has grown through automated documentation platforms, plugins, and audit tools, the author argues that this reflects hype rather than demonstrated demand, especially since major AI providers have not said they consume others’ files. The more durable contribution of the proposal has been clean, predictable Markdown versions of documentation pages, which coding agents frequently use because they are token-efficient and avoid interface clutter; the specification itself has shifted toward helping agents locate those Markdown pages. The author’s organization still generates llms.txt at no cost but prioritizes improved Markdown, agent feedback mechanisms, MCP servers for source-controlled and access-aware documentation retrieval, and service descriptions such as MCP and A2A cards. The piece concludes that organizations seeking AI readiness or governance should focus on usable content formats, direct agent integrations, licensing and legal clarity, while treating llms.txt as a low-priority file that may also require review for potential prompt-injection risks.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.