Company
Date Published
Author
Olivia Greene, Dan Rosanova, Ahmed Saef Zamzam, Prabha Manepalli, Weifan Liang
Word count
2144
Language
English
Hacker News points
None

Summary

Apache Kafka is the de facto standard for event streaming, allowing scale at a level previously unobtainable due to low cost of data streaming. However, expanding without downtime in distributed systems is challenging and requires hard work. Confluent Cloud automates infrastructure management, leveraging cloud provider services such as AWS, Azure, or GCP. The service uses the concept of CKUs (Confluent Units), which encapsulate virtual machines, disks, storage, and other resources into a single package. Clusters can be composed of 1 to N CKUs, with Confluent Cloud placing partitions intelligently in an attempt to minimize downtime during scaling operations. When scaling, Confluent Cloud automates the rebalancing process, moving partitions between brokers while ensuring minimal disruption. This allows for elastic scaling without downtime, making it a seamless experience for event streaming software as a service (SaaS).