Rewinding the exponential: How we keep hardware usage down

Cloudflare’s Data team operates a global pipeline that transports data for products including Logs, SQL API, observability, and internal analytics at roughly two billion events per second, while addressing exponential traffic growth with comparatively linear hardware expansion. To reduce bandwidth, compute, memory, and storage demands, the team first replaced a monolithic schema with product-specific schemas in 2022, reducing backbone traffic by 35–40%, and switched from LZ4 to zstd compression in 2024 for a further nearly 40% reduction without significant CPU impact. In 2026, it redesigned analytics replication so events traverse wide-area networks only once before later replication, expanded sampling into edge-side Logfwdr services to reduce high-volume records while preserving representative estimates, and moved aggregations earlier and repeatedly through the pipeline for datasets that support exact rollups. These changes saved about 70 GB/s of compressed backbone traffic across load balancing, sampling, and aggregation, reclaimed substantial receiver resources, delayed projected throughput growth by roughly eight months, and are expected to halve the pipeline’s projected data volume by 2031.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.