Shared Dictionaries: compression that keeps up with the agentic web

Web pages have been increasing in size by 6-9% annually over the past decade due to their richer, more interactive content, a trend that is not expected to change. However, the frequency of page rebuilds and requests, driven by automated agents, is increasing significantly, leading to challenges in efficient caching and bandwidth usage. Shared compression dictionaries offer a solution by allowing browsers and servers to communicate only the differences in files rather than re-downloading entire files, which results in a significant reduction in data transfer sizes. Cloudflare is developing support for this technology, which involves compressing new file versions against previously cached ones, thereby minimizing redundant data transfer. This approach can notably reduce bandwidth requirements and improve page load times by leveraging compression dictionaries that are specific to each client-server interaction. The adoption of shared dictionaries has been slow due to previous security flaws and technical complexities, but with the introduction of new standards like RFC 9842 and support from major browsers, the infrastructure for widespread implementation is being built. Cloudflare's phased rollout aims to make shared dictionary compression accessible and manageable for all users, thereby enhancing web performance and efficiency in the face of increasing demands from both human and automated traffic.