June 2018 Summaries

Database storage engines, such as B-tree and LSM, have different designs that optimize for either read or write performance, and also consider consistency, transactions, concurrency control, compaction, and compression. Monolithic databases prioritize strong consistency and ACID transactions, while distributed databases like NoSQL databases often sacrifice consistency for availability. Concurrency control mechanisms, such as MVCC, play a crucial role in handling concurrent requests, with SQL databases typically supporting Snapshot Isolation and Serializable levels, while NoSQL databases like Cassandra and DynamoDB use eventual consistency and last-writer-wins semantics. Compaction and compression are also important aspects of database storage engines, with LSM engines offering multiple compaction strategies and compression algorithms, and B-tree engines susceptible to fragmentation that hinders compression. Understanding these factors is essential for a complete understanding of the design of database storage engines.

A busy developer needs to understand the underlying database storage engine when evaluating operational databases, focusing on flexibility, consistency guarantees, scalability, fault tolerance, latency, throughput, and manageability. Two popular types of storage engines are B-tree based and LSM tree based, each with its pros and cons. B-trees prioritize read performance but struggle with write performance due to maintaining a well-ordered data structure. In contrast, LSM trees focus on sequential writes for high-performance workloads with large data volumes, leveraging tiered storage mechanisms. While LSM engines exhibit higher CPU resources during read operations, these issues are mitigated in practice. Both types of engines have been adopted by popular databases and NoSQL systems, including Oracle DB, MS SQL Server, MongoDB, Apache Cassandra, Elasticsearch, and InfluxDB.

Yugabyte has raised $16 million in funding led by Dell Technologies Capital and Lightspeed Venture Partners to expand its reach to large enterprises. The company aims to solve the challenge of cloud-native, high-performance distributed SQL database platforms with its open-source YugaByteDB platform. This platform enables organizations to standardize on a single, non-proprietary database for various workloads requiring distributed SQL capabilities, resulting in reduced operational complexity and costs. With the additional funding, Yugabyte plans to accelerate investments in engineering, sales, and customer success to support enterprises building business-critical applications in the cloud. The company's goal is to democratize the transactional database layer by offering a single non-proprietary database that can span public and private clouds, providing Fortune 500 CIOs with an alternative to cloud vendor lock-in at the database layer, while also benefiting developers who don't need to re-architect existing applications.

The adoption of containerized stateful services, such as databases and message queues, is increasing, with nearly 40% of respondents in the 2018 Kubernetes Application Usage Survey running databases using Kubernetes. This trend has been growing over the last couple of years, driven by the maturity of persistent volumes and stateful workload APIs in non-production environments. The most popular SQL and NoSQL databases are PostgreSQL, MongoDB, Redis, and Elasticsearch, but none were architected for the reliability and dynamic nature of containers. Container orchestration wars have ended with Kubernetes being crowned the de facto king, while Docker Swarm continues to remain relevant for smaller cluster sizes. New management products from Docker and Mesosphere aim to address mid-to-large enterprise needs, and the big 3 cloud platforms are doubling down on their managed Kubernetes offerings. Cloud native databases are emerging, empowering organizations to build and run scalable applications in modern environments, with only a few open source and ACID compliant options available, including Vitess, CockroachDB, FoundationDB, TiDB, and YugabyteDB.