April 2021 Summaries

Incremental Compaction Strategy (ICS) has evolved with the introduction of the Space Amplification Goal (SAG) in ScyllaDB Enterprise 2020.1.6, designed to optimize both space and write performance for overwrite-intensive workloads. Initially, compaction strategies like Size Tiered (STCS) and Leveled (LCS) had limitations, with STCS favoring write performance but suffering from space and read amplification, while LCS excelled in space efficiency at the cost of write amplification. ICS addressed some of these issues but still faced challenges with space amplification under overwrite conditions. The new SAG feature allows ICS to maximize disk utilization without compromising write performance by implementing a hybrid approach that merges the best aspects of both leveled and size-tiered structures, particularly optimizing the largest tier for space efficiency. This makes ICS + SAG, or ICS 2.0, an attractive option for users dealing with overwrite workloads who prioritize space utilization. The configurable SAG parameter enables users to balance space and write amplification according to their specific needs, offering a more efficient alternative to LCS for certain workloads.

ScyllaDB has introduced a shard-aware port to enhance the connection efficiency to its database nodes, supplementing the traditional port by default. This new port, set at 19042 by default, allows drivers to directly connect to specific shards, significantly reducing the number of connection attempts needed, particularly when multiple clients connect simultaneously. The shard-aware port overcomes limitations associated with the older connection method, where drivers could not specify which shard should handle the connection, leading to uneven distribution and increased latency. By using an algorithm based on the connection's source port modulo the number of shards, the shard-aware port streamlines the process, ensuring more consistent and efficient connections. Experimental tests demonstrated that using a shard-aware port significantly reduces the number of connection attempts, thus optimizing performance. However, some limitations exist, such as potential network configuration issues and challenges with client-side NAT, which could necessitate fallback to the non-shard-aware port. The feature is currently supported by some ScyllaDB drivers, with plans to expand support to others in the future.

Coordinated Omission (CO) is a critical issue in performance benchmarking, often leading to misleadingly positive results by failing to accurately measure system outliers and request delays. Coined by Gil Tene, CO occurs when the measuring system unintentionally synchronizes with the system being tested, thereby neglecting to capture latency spikes or missed requests. The text explains CO using an analogy of delayed coffee runs causing a backlog, and emphasizes its prevalence in open versus closed model systems, such as web applications and assembly lines. The text outlines various solutions to mitigate CO, including queuing and queueless request scheduling, and latency correction or simulation techniques. It highlights the importance of setting explicit throughput targets and worker thread numbers to ensure accurate benchmarks. The document also provides practical guidance on using tools like YCSB and ScyllaDB for testing, stressing the need for latency correction to reflect true system performance accurately.

FullContact transitioned from Cassandra to ScyllaDB for their Resolve product, aiming to overcome performance limitations with Cassandra, such as failing to achieve desired query rates and encountering timeouts under heavy loads. Initially attracted to Cassandra for its scalability and low latency, the team faced challenges despite extensive tuning efforts, including compaction strategies and heap size adjustments, which only marginally improved performance. ScyllaDB, a C++-based drop-in replacement for Cassandra, promised significant performance enhancements and successfully delivered, allowing FullContact to exceed their goal of 10,000 queries per second and even peak at over 20,000 requests per second with minimal latency. The switch to ScyllaDB not only resolved performance issues but also simplified maintenance with out-of-the-box performance improvements and easy monitoring integration. Despite initial setbacks with data loading due to format compatibility, FullContact adjusted their data processing strategy to leverage streaming inserts, further optimizing their database operations and significantly enhancing their system's efficiency.

ScyllaDB's DynamoDB-compatible API, known as Alternator, requires a load-balancing solution to effectively distribute client requests across the database's nodes, as clients are not inherently aware of the cluster's structure. The article discusses both server-side and client-side load balancing solutions, ultimately recommending the client-side approach for its simplicity and efficiency. While server-side load balancing, such as using DNS or TCP load balancers, can add complexity, cost, and latency, client-side load balancing involves minimal changes to the client application itself. This method leverages a small library to allow applications to use the AWS SDK as usual while dynamically routing requests to various nodes, thereby ensuring high availability and balanced loads without overcomplicating the ScyllaDB infrastructure. This approach facilitates an easier transition for applications from DynamoDB to Alternator, maintaining functionality with minimal adjustments.

ScyllaDB has introduced a Cloud Calculator designed to assist users in estimating costs based on database needs while highlighting that effective capacity planning goes beyond basic calculations, especially in stateful systems like databases. The tool offers recommendations for server types and costs, considering factors such as read/write operations and data sizes, and allows comparisons with other NoSQL databases like Amazon DynamoDB and DataStax Astra. However, it also has limitations, as it doesn't account for specific CPU or IO needs or advanced features like Workload Prioritization, and assumes a default replication factor. An example use case with the fictional BriefBot.ai illustrates how the calculator can be applied, emphasizing the importance of understanding database models and workload estimations for effective scaling and cost management. The article underscores the critical nature of capacity planning, suggesting that while the calculator provides a useful starting point, ongoing monitoring and adjustments are essential for maintaining efficient and resilient database operations.

ScyllaDB's new IO Scheduler, introduced in the ScyllaDB Open Source 4.4 release, addresses the challenge of balancing disk input/output (IO) operations to optimize performance while minimizing latency. The new scheduler enhances the previous setup by allowing all shards to perform their own IO tasks independently, avoiding overburdening a single CPU and adhering to the principle of dispatching as much as the disk can handle. This is achieved through the introduction of IO groups, which facilitate fair capacity allocation among shards and prevent static disk partitioning that could limit performance. The new scheduler employs a "capacity rovers" algorithm, akin to TCP's congestion control, to manage request dispatching efficiently without the drawbacks of earlier methods. This improvement is particularly beneficial for systems with imbalanced workloads and opens up possibilities for additional features like IO cancellation. Users can test its performance using the io_tester tool in the Seastar repository, and the new scheduler is expected to deliver more uniform latency distributions compared to its predecessor.

In March 2021, ScyllaDB released its Open Source 4.4 version as part of Project Circe, which aims to enhance the database's durability, stability, elasticity, and performance. Key improvements include an upgraded Seastar I/O scheduler that allows for better resource allocation across shards, enabling each shard to utilize the entire disk if necessary. Additionally, shard-aware drivers for languages like Java and Go have been updated to support Change Data Capture (CDC), with plans to extend this feature to other drivers such as Python and Rust. Manageability enhancements include the introduction of a ScyllaDB Monitoring Advisor and a new API for ScyllaDB Manager, which allows users to set maintenance windows. Performance optimizations have also been implemented, such as reducing read amplification and improving the efficiency of repair processes, compaction strategies, and latency reduction methods. These updates are designed to create a more robust and efficient database experience for users.