Give AI Agents the Markdown They Actually Want

AI crawlers often request web pages as HTML, which they must then strip down to plain text, a process that can be inefficient and lossy, particularly for tables and code blocks. To address this, a small JavaScript service running on Fastly Compute has been developed to serve Markdown versions of web content specifically to AI crawlers, while normal browser requests continue to receive HTML. This service, which requires about 200 lines of JavaScript, uses tools like linkedom and Defuddle for parsing and extracting content, and Turndown for converting HTML to Markdown. The approach reduces bandwidth and improves the accuracy of AI models by providing them with cleaner, more digestible content. The service includes caching to optimize repeated requests and can be customized for specific needs like link rewriting or site-specific content extraction. This transformation process respects the efficiency needs of AI crawlers and optimizes content delivery without disrupting the traditional user experience.