Company
Date Published
Author
Lucia Cerchie, Yeva Byzek, Josep Prat
Word count
1399
Language
English
Hacker News points
None

Summary

Apache Kafka is moving away from its dependency on ZooKeeper, a distributed configuration service, with the introduction of KIP-500, which includes a built-in consensus layer in Apache Kafka 2.8.0. This change simplifies infrastructure design and operational workflows for Kafka deployments by removing the need to administer an additional system. By removing ZooKeeper, users can stop worrying about learning another distributed system, administering separate servers or VMs, and managing security configurations. Additionally, this change reduces storage considerations, as Kafka no longer requires capacity planning for ZooKeeper. The new consensus layer in KIP-500 improves control plane traffic, reducing the time it takes to read metadata from ZooKeeper and improving operations. This change also eliminates the need to monitor ZooKeeper metrics and troubleshoot issues related to divergent state between the controller and ZooKeeper. Users can now focus on core components of Kafka without the added complexity of managing ZooKeeper.