Scaling Security Insights: how we achieved a 10x increase in global scanning capacity

Cloudflare's Security Insights faced challenges in effectively scanning for security risks due to infrequent scans and an opt-in system that left many accounts unchecked, amidst a landscape of increasing automated attacks. To address this, Cloudflare sought to boost its scanning throughput tenfold, from 10 to 100 scans per second, all while dealing with system load issues such as API timeouts and crashing processes. Key enhancements included optimizing Apache Kafka's message consumption through parallel processing, resolving head-of-line blocking with a 'fast lane' and 'slow lane' system, and refining database query strategies to reduce latency. The company also rethought its scheduling mechanism to ensure a balanced distribution of scans, which involved scheduling zones independently of accounts and implementing adaptive rate limiting. These measures not only achieved the targeted tenfold increase in scan throughput, reaching over 120 scans per second during peak times, but also allowed for more frequent and automated scans across all account types, thus enhancing system stability and enabling new features like granular on-demand scans. This success story underscores the importance of a deep understanding of existing systems and metrics in solving scalability issues without resorting to simply adding more resources.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.