The blog post examines the transition from using ZooKeeper to an internal log for Apache Kafka's metadata management, highlighting the scalability and efficiency challenges faced with the traditional ZooKeeper-based system. The article discusses the development and implementation of KRaft, a modified Raft consensus protocol designed to enhance Kafka's metadata management by utilizing quorum replication instead of the primary-backup algorithm. This shift allows Kafka to handle metadata changes more efficiently, reducing bottlenecks and improving performance during broker shutdowns and controller failovers. The new system introduces the Quorum Controller, which operates without ZooKeeper and leverages the KRaft protocol to maintain consistent and synchronized metadata logs across brokers. The Quorum Controller's design significantly reduces latency in operations like broker shutdowns and controller failovers, as demonstrated in experiments with two million topic partitions. The post also announces the release of Apache Kafka 3.8.0, which includes these improvements and other new features.