Company
Date Published
Author
Sophie Blee-Goldman, Victoria Xia, Wade Waldron
Word count
3413
Language
English
Hacker News points
None

Summary

The Apache Kafka rebalancing protocol is designed to ensure that consumer groups can rebalance their partitions without causing downtime or significant performance degradation. The current eager rebalancing protocol, which was introduced in version 2.3 of Kafka Streams, stops the world and forces all consumers to revoke their partitions before rejoining the group. However, this approach has drawbacks, including downtime for all partitions and all members of the group, as well as potential issues with partition ownership and migration. To address these challenges, a new incremental cooperative rebalancing protocol was introduced in version 2.4 of Kafka, which allows consumers to continue processing records while reassigning their partitions incrementally. The new protocol is designed to be more efficient and less painful than the eager protocol, while still maintaining at-least-once and exactly-once processing guarantees. By using a cooperative partition assignor, such as the CooperativeStickyAssignor, consumers can ensure that their partitions are reassigned safely and efficiently, without causing downtime or performance degradation. The new protocol has been extensively tested in Kafka Streams and ksqlDB applications, and has shown promising results in reducing rebalance times and improving overall availability.