How Palo Alto Networks Replaced Kafka with ScyllaDB for Stream Processing

Palo Alto Networks, a global leader in cybersecurity, processes terabytes of network security events daily and sought a solution to correlate these events in near real-time without the operational overhead of deploying an additional message queue system like Kafka. The engineering team opted to replace Kafka with ScyllaDB, their existing low-latency distributed NoSQL database, to serve both as an event data store and a message queue. This approach allowed them to streamline operations by eliminating Kafka, reducing costs, and maintaining high throughput performance. The system designed by Principal Software Engineer Daniel Belenky and his team involves the ingestion of disparate events from various sensors, normalization of data into a canonical form, and the use of ScyllaDB to shard the data for parallel processing by multiple worker components. This architecture succeeded in achieving the project goals while minimizing complexity and operational costs, highlighting the potential of ScyllaDB for other organizations facing similar challenges in stream processing and event correlation.