Making agent-friendly pages with content negotiation

Agents that browse the web require clean, structured text rather than the CSS, client-side JavaScript, and images that humans typically view, as these elements fill up their context window and consume tokens without providing useful information. To address this, a new update has been implemented that allows content to be delivered in markdown for agents while maintaining a full HTML and CSS experience for human readers. This is achieved through content negotiation, where the HTTP Accept header specifies the preferred format, and agents like Claude Code signal a preference for markdown by listing it first in their requests. A middleware checks these preferences and routes markdown-preferred requests to a Next.js route handler, converting rich-text content to markdown while preserving its structure, ensuring that code blocks, headings, and links remain functional. The markdown version significantly reduces payload size, allowing agents operating under token limits to consume more content efficiently. Synchronization between HTML and markdown versions is maintained using Next.js 16 remote cache and shared slugs, ensuring that both versions refresh simultaneously when content updates occur. Additionally, a markdown sitemap helps agents discover available content by listing metadata about each piece, including publication dates, content types, and links to both HTML and markdown versions, enabling agents to select the format best suited for their needs.