A database for 2022

Faced with limitations from using etcd for their database, the team opted to transition to SQLite with the aid of Litestream, a tool that facilitates live replication by streaming SQLite's WAL file changes to S3. This approach allows near real-time backups without modifying SQLite, providing a more scalable and manageable solution. The migration process involved creating a main SQLite database for high-value data and a separate one for ephemeral data, with schema changes made to better suit their needs. The transition from etcd was executed in stages, initially writing data to both systems before fully switching to SQLite. Despite some challenges with schema migrations and the need for extensive service code modifications, SQLite proved effective, with tuning required under load conditions. The team anticipates further improvements, including setting up read-only Litestream replicas, as they continue refining their database architecture to alleviate past complexities and enhance scalability and ease of use.