Company
Date Published
Author
Wade Waldron, Victoria Xia, Michael Drogalis
Word count
2906
Language
English
Hacker News points
2

Summary

In ksqlDB, distributed systems give scale and fault tolerance, allowing for linear scalability even in the presence of faults. When scaling up, each server processes only a fraction of the total work, making it more efficient and reducing resource waste. The architecture scales by dividing the workload among multiple servers, with each server running all submitted queries. Fault tolerance is achieved through two categories of operations: stateless and stateful. Stateless operations are easy to recover from as they don't require memory, while stateful operations require memory and can be slow to recover from due to replaying changelog data. However, ksqlDB can achieve high availability without downtime by eagerly replaying changelog data, making failures invisible to users. The system is designed around Apache Kafka Streams and RocksDB, abstracting task-level processing and hiding the details of how it works behind configuration parameters.