Company
Date Published
Author
Tom Schreiber and Derek Chia
Word count
3672
Language
English
Hacker News points
218

Summary

ClickHouse Keeper is a fast, resource-efficient, and feature-rich alternative to ZooKeeper that provides a reliable metadata store and coordination mechanisms for real-time applications and analytics. It offers several benefits over ZooKeeper, including easier setup and operation, better compression, no limit on packet and node data size, faster recovery after network partitions, additional consistency guarantees, and more resource efficiency. ClickHouse Keeper was originally developed for use with ClickHouse when deployed as a distributed system in a self-managed setup or hosted offering like CloudHouse Cloud, but its broader community benefits are now being explored. The development of ClickHouse Keeper has been production-ready for over 1.5 years, and it has been deployed at scale in various environments, including ClickHouse Cloud. ClickHouse Keeper provides coordination systems for data replication, automatic insert deduplication, and consensus mechanisms for part names and assigning part merges to specific cluster nodes. It is used in various ClickHouse applications, such as task scheduling queues, Kafka Connect Sinks, and replicated database engines. The benchmarking results show that ClickHouse Keeper consistently uses significantly less memory than ZooKeeper for the same volume of data while maintaining performance close to ZooKeeper. Looking forward, ClickHouse Keeper plans to introduce a multi-group Raft protocol for better support of scenarios with high parallel read/write requirements.