September 2021 Summaries

The blog post by Michał Chojnowski delves into the process of troubleshooting a performance issue encountered when running ScyllaDB on Oracle Cloud's ARM-based Ampere A1 servers. Despite the initial suspicion that the problem was hardware-related, it was discovered that the issue stemmed from a software-level CPU bottleneck, where certain runs of the database exhibited significantly reduced throughput. Through meticulous investigation using various performance monitoring tools, it was revealed that the low throughput was due to cache line invalidations caused by a shared global tree node, leading to substantial performance degradation. The author highlights the complexity of identifying such subtle issues, as the bottleneck was not apparent from traditional sampling methods and required a detailed understanding of the interactions between software and NUMA architecture. This deep dive exemplifies the challenges developers might face when optimizing applications for ARM-based platforms and underscores the importance of considering memory access patterns and their impact on performance.

ScyllaDB Operator 1.5 has been released, offering enhanced stability and new features for running ScyllaDB on Kubernetes, with backward-compatible API changes. This version supports ScyllaDB Open Source and Enterprise users by automating tasks such as installation, scaling, and rolling upgrades of ScyllaDB clusters. Notable improvements include support for specifying image pull secrets for private registries, enhanced resource and placement management, and better high availability through scheduling preferences and readiness probes. The release is compatible with ScyllaDB versions 4.3 and above, ScyllaDB Enterprise 2021.1 and above, and Kubernetes version 1.19.10 and above. Users can upgrade from version 1.4.x without additional steps using helm or kubectl, and further details are available in the GitHub release notes and ScyllaDB Operator documentation.

Michał Chojnowski's blog post analyzes the performance of ScyllaDB on AWS's Graviton2-based Arm instances compared to Intel's x86-based instances, highlighting a 15%-25% better price-performance ratio for Arm across both CPU-bound and disk-bound workloads with similar latencies. The comparison involved m5d.8xlarge and m6gd.8xlarge instances, which have similar specifications except for the CPU architecture. The benchmarks conducted using the cassandra-stress tool revealed that while general performance was roughly equivalent, the cost-efficiency of Arm provided a significant advantage. Despite some slower random mixed read-write operations on Arm, the overall price-performance benefit was about 20% in favor of m6gd. The blog post concludes that AWS Graviton2 ARM-based servers are competitive with or superior to their x86 counterparts, especially in terms of price-performance, and it anticipates future official ScyllaDB releases for Arm instances.

The blog post by Pavel "Xemul" Emelyanov discusses the intricacies of IO scheduling in ScyllaDB, emphasizing the importance of prioritizing IO requests to ensure timely processing, especially in latency-sensitive OLTP workloads. ScyllaDB utilizes the Seastar framework's IO scheduler, which manages request priorities and dispatches tasks into multiple queues, ultimately optimizing them for processing by the Linux kernel and disk. The scheduler's critical parameter, the "latency goal," guides the system in managing concurrency levels to ensure requests are completed within a specified time frame, despite challenges posed by varying disk performance characteristics. Tools like iotune are employed to measure disk performance parameters in advance, aiding in accurate scheduler configuration. The post also delves into handling "pure," "mixed," and "unstable" workloads, highlighting how request size and type affect disk throughput and detailing techniques for maintaining performance under different conditions. The article underscores the complexity of modeling disk behavior due to its unpredictable nature and emphasizes the need for a nuanced approach to configuring IO schedulers for optimal performance.

OlaCabs, a prominent mobility platform operating in over 250 cities worldwide, including India, Australia, New Zealand, and the UK, leverages ScyllaDB's NoSQL database to efficiently manage their ride-sharing services. Starting in 2016, OlaCabs adopted ScyllaDB to address challenges posed by spiky intraday traffic, opting for a high-availability system over traditional RDBMS due to its need for high throughput and low-latency operations. OlaCabs' architecture integrates Apache Kafka for data streaming, Apache Spark for machine learning, and ScyllaDB as their real-time operational data store, facilitating data synchronization between passengers and drivers by analyzing historical traffic patterns and behavior. Anil Yadav, Engineering Manager at OlaCabs, shared insights on ScyllaDB's production use at the ScyllaDB Summit 2021, highlighting how OlaCabs employs machine learning and analytics for both operational efficiency and business growth.

ScyllaDB has been explored for its capabilities in distributed numerical calculations, particularly with large matrices, by using it as a distributed cluster that functions like RAM, enabling operations on matrices that wouldn't fit in traditional storage. The project focused on implementing a library for performing linear algebra computations using ScyllaDB, leveraging the Basic Linear Algebra Subprograms (BLAS) interface to facilitate integration with existing numerical codes. The researchers opted for a block-based representation of matrices, which allows efficient parallel computations by dividing tasks among multiple workers connected to ScyllaDB. This approach enhances scalability and concurrency, as demonstrated in benchmarks performed on AWS, which showed efficient matrix-matrix multiplication. The project also involved developing a modern C++ wrapper for the ScyllaDB driver to improve usability. The research, conducted with the University of Warsaw, resulted in a scalable solution for sparse computational algebra, with code available for public access on GitHub.

Deploying the ScyllaDB Operator on Kubernetes clusters involves various methods, including using GitOps practices and Helm charts, to manage infrastructure and application configurations. ScyllaDB Operator provides support for different deployment strategies, with versions 1.1 and 1.2 introducing Helm charts and publishing deployment manifests for manual or GitOps automation, respectively. GitOps employs Git as the single source of truth, making system states visible and auditable, with Kubernetes manifests stored in a Git repository. The deployment process involves setting up local storage using Kubernetes persistent volumes and installing necessary components like the cert-manager for internal webhooks. The deployment manifests are organized into specific folders and need to be applied in sequence due to interdependencies, ensuring components such as scylla-manager and ScyllaDBCluster are established. Customization is possible through Git, allowing adjustments to manifest resources without compromising supported deployments. The ScyllaDB team is working to simplify deployment further and plans to introduce an Operator Lifecycle Manager bundle on operatorhub.io.

ScyllaDB's Project Circe, an initiative aimed at enhancing ScyllaDB's consistency and performance, introduced several updates in August 2021, including the new Safe Mode to guide users away from non-recommended options in production. The project compares ScyllaDB's performance with Apache Cassandra and advances the integration of Raft for stronger consistency. ScyllaDB's development team dedicates 20% of their time to personal projects, such as adding WebAssembly to user-defined functions. Recent updates include enabling repair-based node operations by default, implementing User-Defined Aggregates, optimizing memory-related diagnostics, and improving row cache behavior. The installer now supports RAID 5, and the docker image base has switched to Ubuntu 20.04. ScyllaDB's Monitoring Stack continues to evolve, with the addition of a Consistency Level Calculator and updates to ScyllaDB Manager and Operator.