Company
Date Published
Author
Russell Brown
Word count
2095
Language
English
Hacker News points
None

Summary

The Ditto Big Peer Store is a distributed database system designed to scale dynamically while maintaining availability and consistency. It consists of various components, including Subscription Servers, API Servers, a distributed transaction log, and a partitioned, replicated database known as the Big Peer Store. The system handles data as a distributed database, storing documents as Conflict-free Replicated Data Types (CRDTs) and employing random slicing for partitioning, allowing for balanced distribution and minimal data movement during scaling. Replication ensures data redundancy, while the Transaction Log orders transactions, assigning each a unique timestamp for consistency. The system uses a Universally Stable Timestamp (UST) for ensuring consistent reads across nodes, and a Garbage Collection Timestamp to manage data storage efficiently. Scaling involves transitioning between cluster configurations, with processes like backfill and gossip ensuring data consistency during reconfiguration. This transition process is governed by transaction timestamps and cluster configuration epochs, allowing the system to adapt to changes in size and structure while maintaining transactional causal consistency.