The Current State of Content Negotiation for AI Agents (Feb 2026)

The text emphasizes the increasing importance of AI agents in web interactions and the inefficiency of traditional HTML for these agents due to its token-heavy nature, which can hinder performance and increase costs. It proposes content negotiation via the HTTP Accept header as a solution, allowing agents to request lighter, more efficient markdown versions of web content. The text highlights that while some AI agents like Claude Code and Cursor are already using this method to request markdown, others like OpenAI Codex are not, leading to unnecessary token consumption. Furthermore, it introduces the idea of llms.txt files and agent skills as additional strategies to enhance AI agent efficiency by directing them to relevant content and providing them structured guidance. The document also outlines future improvements in developer platforms, like Checkly, to better accommodate AI agents by optimizing CLI outputs and API responses for markdown, thus fostering an agent-friendly web environment.