Company
Date Published
Author
Lucia Cerchie, Josep Prat, Yasuhiro Matsuda
Word count
1987
Language
English
Hacker News points
None

Summary

Apache Kafka's "request purgatory" is a mechanism designed to efficiently manage large-scale asynchronous operations by holding requests until their criteria are met or a timeout occurs. The original implementation faced issues such as potential memory exhaustion due to the delayed deletion of completed requests, which led to performance bottlenecks. A new design using Hierarchical Timing Wheels and a doubly linked list offers improved efficiency with immediate O(1) deletion of completed requests and reduced load from purge operations. This design enhances scalability and CPU usage, as demonstrated by benchmark tests showing a significant increase in request handling capacity compared to the old version. The upgrade aligns with Apache Kafka's continued evolution, as evidenced by the release of Kafka 3.8.0, which introduces various new features and improvements.