August 2017 Summaries

Facebook's original database architecture was built using MySQL as the persistent database and memcache as a "lookaside" cache, but this proved to be unsuitable for the company's growing user base. In response, Facebook developed TAO, a custom NoSQL graph API built on top of sharded MySQL, which solved many of the problems faced by the company, including data consistency issues and loss of developer agility. However, this approach required significant engineering effort and led to a loss of flexibility for developers. Meanwhile, Google was building Spanner, an entirely new SQL database that could horizontally scale, geo-replicate, and tolerate infrastructure failures. Unlike TAO, Spanner uses per-shard distributed consensus to ensure high availability in the presence of failures. Today, companies like YugabyteDB are building on top of the Spanner architecture to bring similar benefits to enterprises, including developer agility, global data distribution, and extreme resilience against failures.

The author, a seasoned developer with experience working on distributed systems, shares their perspective on the state of the database world in 2017, highlighting the challenges faced by developers when choosing between NoSQL and SQL databases. The author notes that the initial focus on NoSQL was misguided, as it prioritized language features over production-grade stability and consistency. Instead, the author advocates for "non-relational SQL" databases that combine the benefits of SQL with those of NoSQL, allowing for scalable and reliable data storage and retrieval. Today, sharded SQL databases are the preferred choice for mission-critical applications, while NoSQL databases are used to handle large datasets or provide faster data access. The author also discusses the limitations of clustering solutions, which can be combined with manual sharding to achieve scale out and HA, but require significant complexity to implement.