Linearizable Distributed Map On Kafka

Implementing a distributed key-value (KV) store on top of Kafka involves leveraging consensus algorithms and the total order broadcast concept to achieve linearizable storage, which ensures consistent and reliable data management. The blog post explores converting Kafka into a replicated linearizable KV store, emphasizing the need for a fault-tolerant and consistent system for the SchemaRegistry project without introducing additional failure points. It details achieving linearizable writes and reads by using versioned values and appending dummy messages to the log, respectively, and highlights the challenges posed by the CAP Theorem, where consistency is prioritized over availability in case of network partitioning. The post also addresses potential issues like the infinite log, suggesting Kafka's log compaction to retain only the latest values and using configurations to manage message retention. Lastly, the prospect of implementing other Java data structures with varying consistency guarantees is proposed for further exploration.