Docs URL Benchmark: Markdown & llms.txt > HTML

Browserbase encountered a persistent issue where Claude Code generated 404 errors when accessing Mintlify-hosted documentation, due to the absence of navigational guidance in stripped-down markdown pages meant for AI agents. To address this, a simple solution involved linking to an llms.txt file, which serves as a comprehensive map of available pages, dramatically reducing dead URLs and unnecessary fetches. A benchmark test, involving 2,400 runs across different document formats and AI models, confirmed that linking to llms.txt was the most efficient method, cutting down token usage and improving navigation without compromising accuracy. The study highlights the growing importance of structuring documentation not just for human readability but also for the navigation capabilities of AI agents, with the llms.txt link proving to be an effective and low-cost solution for optimizing agent interactions with documentation.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.