How a Tier‑1 Bank Tuned Apache Kafka® for Ultra‑Low‑Latency Trading

A global investment bank collaborated with Confluent to achieve ultra-low latency in their trading pipelines, reaching a sub-5ms, 99th percentile latency at a rate of 1.6 million messages per second, crucial for real-time trading in global capital markets. This was accomplished through a meticulous approach involving architectural discipline, comprehensive monitoring, and strategic configurations in a multi-data center deployment, focusing on every aspect of the Kafka message path to mitigate latency outliers. The project emphasized the significance of understanding single-partition latency baselines, addressing infrastructure bottlenecks, and employing a scientific, iterative tuning process. Utilizing tools like the OpenMessaging Benchmark, the team was able to systematically enhance system performance, ensuring robust order guarantees and high throughput, which are vital for mission-critical financial applications. This case study provides valuable insights into the challenges and best practices for achieving low-latency streaming with Kafka at scale, highlighting the importance of infrastructure upgrades, such as enterprise SSDs and ZGC, and the role of reproducible benchmarking in performance optimization.