August 2018 Summaries

The Yugabyte Database team has released updates, including a new version of the database (1.0.5) that adds Cassandra-compatible YCQL features and Redis-compatible YEDIS features such as authentication support using passwords and safety flags to disable dropping data inadvertently. The team also announced support for Google Container Engine (GKE) and compared YugabyteDB with other Google Cloud databases like Spanner. Additionally, the team is hiring a Developer Advocate and has released new documentation and blog posts on various topics related to YugabyteDB.

YugabyteDB can be effectively run on top of Kubernetes, leveraging its automated sharding and strongly consistent replication architecture. The database's components are modeled as independent StatefulSets, allowing for efficient management and scaling in a containerized environment. To deploy YugabyteDB on Kubernetes, users need to create YAML files that define the YB-Master and YB-TServer StatefulSets, including configuration parameters such as replica counts, pod anti-affinity rules, and volume mounts. Running a 4-node YugabyteDB cluster on minikube provides a simple way to demonstrate the database's capabilities in a local environment. The cluster can be scaled horizontally by adding or removing nodes, and vertically by upgrading the container image, all while maintaining fault tolerance and high availability.

YugabyteDB has successfully tested a highly dense cluster configuration where each node stores up to 4.5 TB of data, exceeding the recommended limit for DataStax Enterprise-compatible databases like Cassandra. The cluster was built on general purpose SSDs with lesser RAM than previously used high-end instance types, and it achieved high performance metrics, including 19K reads/sec across four nodes at 1.3ms latency, and 20-25K write IOPS at about 3-4ms latency. The cluster demonstrated reliable and rapid scaling capabilities when adding a new node to the existing cluster, with the new node becoming operational within hours despite handling large amounts of data from the other four nodes. This achievement showcases YugabyteDB's ability to handle high data density while maintaining performance and reliability, making it an attractive option for organizations seeking cost savings through optimized database configurations.

Apache Cassandra is an open-source distributed database that can handle "big data" workloads by distributing data and reads across multiple nodes without a single point of failure. It uses a Log Structured Merge (LSM) storage engine, which includes a commit log, memtable, SSTables, and compaction processes to ensure data integrity and performance. Cassandra is written in Java and has limitations due to its garbage collection issues, which can lead to performance problems. In contrast, YugabyteDB is implemented in C++ with no need for garbage collection, offering better throughput and predictable response times. Cassandra prioritizes availability and partition tolerance over consistency, while YugabyteDB ensures stronger consistency and ACID transactions. Lightweight Transactions (LWT) are used in Cassandra to provide conditional inserts/updates, but require four round trips, whereas YugabyteDB supports them with just one round-trip. Cassandra's secondary indexes can be beneficial for query performance, but have limitations due to their distributed nature. YugabyteDB provides consistent and ACID-compliant secondary indexes built on its distributed transactions. For multi-region deployments, Cassandra uses LOCAL_QUORUM consistency, while YugabyteDB offers global or regional consistency with continuous availability under region failures or zone/rack failures, respectively.

YugabyteDB utilizes Raft for both leader election and data replication in its distributed SQL database architecture, applying the consensus algorithm at an individual shard level where each shard has its own Raft group. This allows for strong consistency with zero data loss writes, continuous availability even under node/disk/network failures, rapid scale-out and scale-in with auto-rebalancing, high performance with quorumless reads and tunable latency, and a unique combination of automated sharding and Raft-based replication that frees application developers from traditional compromises.

Consensus-based replication is crucial for building resilient and strongly consistent distributed systems. Paxos, first proposed in 1990, is a leader-based consensus protocol that allows participants to agree on values by giving up on other stalled participants after some amount of time. Despite its widespread use, Paxos remains challenging to implement due to its complexity. Raft, introduced in 2013, is a leaderless consensus protocol designed as an alternative to Paxos, offering better understandability and formal proof of safety. With over 100 open-source implementations, Raft has become the de-facto standard for achieving consistency in modern distributed systems, with many popular databases such as YugabyteDB, CockroachDB, and TiDB adopting it for leader election and data replication.

YugabyteDB provides consistent, high-performance secondary indexes built on top of distributed ACID transactions. Secondary indexes can be created using one or more columns of a database table and provide the basis for both rapid random lookups and efficient access of ordered records when querying by those columns. Creating a secondary index requires additional writes and storage space to maintain the index data structure. YugabyteDB uses distributed ACID transactions under the hood in order to maintain consistency of secondary indexes, ensuring that read queries see consistent values even when updates are happening at the same time as reads. Secondary indexes can be used to efficiently look up users by their email address or other columns, and can be used with pre-existing data, allowing for future flexibility.