September 2023 Summaries

In YugabyteDB, there are two main services: the YB-TServer service responsible for all database-level operations and the YB-Master service that maintains cluster consistency by propagating cluster-level operations and storing metadata. The tables and indexes in both services are split into tablets, which are replicated based on the cluster-level replication factor (RF) value, forming RAFT groups with a single leader and multiple followers. A YB-Master server has master-leaders and master-followers that receive, apply, and acknowledge changes, ensuring high availability. The tables are distributed across nodes in the cluster, with each node hosting more than one tablet-leader and tablet-follower for its assigned data.

YugabyteDB has partnered with AWS to provide enhanced security and savings for customers. As an AWS Service Ready Partner for PrivateLink, Yugabyte can now connect its clusters without exposing data to the public internet, meeting the requirements of enterprises in sensitive sectors. Additionally, as an AWS Graviton Ready Partner, Yugabyte achieves better performance-per-price ratio on Graviton instances, reducing cloud costs. The partnership offers flexible networking options, including PrivateLink, VPC peering, and Transit Gateway, allowing customers to choose the best solution for their specific needs.

YugabyteDB's new built-in Connection Manager in version 2.19.2 improves performance and simplifies application architecture by enabling up to 20x faster connection creation times and increasing the number of supported connections up to 5x, while maintaining a constant throughput. This addresses a well-known PostgreSQL weakness, turning it into a strength. The new Connection Manager uses logical connections to guarantee consistent database connectivity for applications and supports SQL features such as TEMP TABLE, WITH HOLD CURSORS, and more, with no limitations compared to other pooling solutions running in transaction mode. It also supports session parameters and prepared statements, overcoming some SQL limitations. YugabyteDB's Connection Manager has been modified from the open source connection pooler Odyssey and is available for technical preview in 2.19.

When a node is lost in a running cluster and then brought back online, the system prioritizes consistency over availability according to the CAP theorem, but still maintains high availability with a replication factor of 3. A 3-second re-election process ensures that a new leader is elected, causing temporary latency for certain operations. After being repaired, the node is caught up by the remaining nodes and leaders are redistributed equally across all nodes. If a node is down for longer than 15 minutes, it will be removed from the system unless replaced with a new node, allowing data to be replicated behind the scenes without manual intervention.

Chaos testing is a critical methodology used to enhance system resilience by simulating real-world failures to identify vulnerabilities within database systems, as demonstrated through a case study on YugabyteDB’s Change Data Capture (CDC) functionality. By creating controlled chaos, this approach helps uncover weaknesses that regular unit tests might miss, allowing for proactive improvements. The testing process involves setting up a robust environment using tools like Docker, employing various chaos scenarios such as server crashes and network disruptions, and conducting long-running tests to observe system behavior under stress. The insights gained from chaos testing have led to significant improvements in system resilience, revealing over 100 issues that were addressed to ensure data integrity, recovery, and performance. Chaos testing not only identifies potential breaking points but also fosters a mindset of continuous improvement and resilience in handling unexpected challenges, ultimately leading to a more reliable and trustworthy database system.

Scaling Sequences with Server-Level Caching in YugabyteDB` explores how to generate unique, sequential numbers efficiently in a distributed database like YugabyteDB. The article discusses the challenges of generating unique values in a multi-application environment and introduces database sequences as a solution. It shows how sequences can be used to prevent duplicate values and provide sequential numbers. However, in a distributed setup, this approach has limitations due to latency issues caused by replication and caching. To address these challenges, YugabyteDB now supports server-level sequence caching, which enables all connections to share the same cached sequence values across specific YB-Tserver nodes. This reduces latencies in multi-region setups and improves performance of applications that rely on database sequences. By using server-side UUIDs or application-side ULIDs when possible and enabling per-server sequence caching, developers can improve the efficiency of their distributed applications.

YugabyteDB Voyager 1.5 introduces powerful new live migration features from Oracle with optional fall-forward capabilities, aiming to simplify and speed up app and data migrations while reducing risks. This release includes a live migration workflow that migrates data behind the scenes without disrupting application availability or performance. The feature also offers a fall-forward approach, which creates a replica database to reduce risk during data migrations. Additionally, Voyager 1.5 introduces new customizations, enhanced security features, and provides a sneak peek at a portfolio assessment capability planned for delivery soon. This tool aims to solve common migration pain points by providing a detailed assessment of applications and databases, offering guidance on migration complexity, and guiding users through the migration process with minimal effort.

YugabyteDB takes a unique approach to the storage layer compared to PostgreSQL, which affects the applicability of certain extensions like pg_repack and pg_buffercache. These extensions are related to the buffer cache and table bloat removal, but YugabyteDB's LSM-tree-based storage engine handles compactions that remove expired data, making these extensions unnecessary.

YugabyteDB Managed has introduced several transformative updates to enhance resilience, monitoring, security, and scalability for its users. The new features include the ability to create multi-region clusters on Microsoft Azure, providing unparalleled reliability and fault tolerance. A unified monitoring experience with metrics export to external observability tools simplifies the process of managing multiple platforms. Secure, enterprise-ready networking options are also available through AWS or Azure PrivateLink. Additionally, higher fault tolerance levels and flexible read replicas allow for more resilient mission-critical applications and cost-effective scaling. The service is currently available as a free trial, enabling users to experience its full benefits without any charge.

Karthik Ranganathan, Yugabyte CTO, recently announced several product enhancements for YugabyteDB that are making PostgreSQL dreams a reality. The new features include bimodal query execution, which allows YugabyteDB to deliver PostgreSQL-like performance while also providing resilience and scale; a cost-based optimizer that can improve query execution time by up to 40%; and pain-free connection management, which simplifies application architecture and accelerates productivity. Additionally, YugabyteDB Voyager 1.5 has been released with features such as live migration for no app disruption, built-in portfolio assessment, and simplified disaster recovery workflows. These updates aim to turbocharge PostgreSQL into the future via innovative new features of YugabyteDB 2.19 and Voyager 1.5, making it easier for organizations to lift and shift large sets of applications from PostgreSQL to a modern distributed database.

You can set different RF values and server counts for the read replica cluster to achieve high availability and low latency reads for users in specific regions. Read replicas are observer nodes that replicate data from the primary cluster asynchronously, but do not guarantee strong consistency or ACID compliance. The replication factor and server count of read replicas are separate from those of the primary cluster, allowing you to customize them according to your needs. For high availability, it's recommended to have a higher node count and replication factor greater than 1 for read replicas.

Introducing PgCompute, a client-side PostgreSQL extension that enables JavaScript functions to be executed directly on the database from application logic, improving developer experience by allowing developers to create and optimize database functions within their preferred IDE and programming language. With PgCompute, users can deploy and manage database functions automatically or manually, with features such as automatic deployment mode and native integration with Supabase. The extension uses DeploymentMode.AUTO by default, which automatically deploys and updates functions in the database as needed, reducing network latency and improving performance. By leveraging PgCompute, developers can work with database functions more efficiently, making it an ideal solution for data and compute-intensive tasks such as traversing and manipulating large datasets.

YugabyteDB is a distributed SQL database designed to overcome the challenges faced by PostgreSQL, including Disastrous Transaction ID (XID) Wraparound, Data Loss Likely During Failover, Inefficient Replication That Spreads Corruption, MVCC Garbage Frequently Painful, Process-Per-Connection = Pain at Scale, Primary Key Index Takes Up A Lot of Space, Major Version Upgrades Can Require Downtime, Cumbersome Replication Setup and Troubleshooting Process, Ridiculous No-Planner-Hints Dogma, and No Block Compression. YugabyteDB addresses these issues by employing a modern storage layer design, Hybrid Logical Time (HLC) for cluster synchronization, efficient sync replication, logical replication, per-column updates, one process per connection, log-structured merge-tree (LSM) with primary index structure, online upgrades, fully automated replication, and compression and encryption at rest and in transit. YugabyteDB is designed to be compatible with PostgreSQL, offering a scalable solution for cloud-native applications that require high availability and performance.

What Happens to a Downed Node When It Comes Back Into the Cluster?` When a node goes down in a YugabyteDB cluster, a re-election process begins for the remaining tablet-peers in the RAFT group whose leader was on the downed node. The physical copy of the data remains on that downed node, and when it comes back up, the data is typically in a different state than the remaining nodes due to the 15-minute re-election period. During this time, the downed node remains part of the RAFT group, but if it resumes operation before the timeout, it will receive all data changes that occurred during the downtime from the remaining nodes. If the node is down for longer than the default 15 minutes, its copies of data will replicate to the remaining nodes, and it will be kicked out of the quorum, potentially impacting full functionality of the cluster if not replaced immediately. Rolling restarts are often performed to apply configuration flag settings that cannot be changed online, which may affect connections to the node going down.