Achieving relentless Kafka reliability at scale with the Streaming Platform

Guillaume Bort from Datadog shares their experience of scaling Apache Kafka to meet the demands of a massive data platform. The company built a custom Streaming Platform to abstract Kafka's complexity, enabling real-time reliability at scale. This platform uses Streams to build resilient pipelines decoupled from specific clusters, an Assigner for dynamic cluster management, and a smarter commit log to overcome traditional Kafka limitations such as head-of-line blocking. A custom client library called libstreaming was developed in Rust to optimize performance and observability across all applications. The Streaming Platform allows Datadog to treat Kafka infrastructure like commodity hardware, modulating workloads across clusters, automatically replacing unhealthy components, and ensuring uninterrupted data flow.