Scaling llms.txt

Mintlify has redesigned its llms.txt generation system to prevent oversized files from overwhelming AI agents with irrelevant documentation, which can increase token costs and slow retrieval. Instead of using a single file capped at 100,000 characters, the new architecture creates a navigation-based hierarchy in which sections such as guides, API references, and integrations are moved into descriptive child files under /_llms/, allowing agents to follow only relevant paths. The generator uses a site’s docs.json navigation structure to identify logical groupings and repeatedly moves the sections that save the most space until each file fits the character limit. In tests across 200 retrieval tasks on ten documentation sites using Claude Sonnet 5 and GPT-5.6 Terra, the hierarchical approach reduced average task completion time by 52.1%, token use by 44.8%, and agent 404 errors by 88.1%. The feature is now available to all Mintlify customers and regenerates automatically whenever documentation is published.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.