March 2021 Summaries

ScyllaDB Open Source 4.4 has been released, offering improved performance, stability enhancements, and bug fixes. This version introduces a new feature for setting timeouts on individual queries, allowing for more granular control over query execution times. It also includes enhancements like additional client information in the system.clients table and improved deployment options, including support for Ubuntu 20 and node-exporter packaging, simplifying installation. This release marks significant performance optimizations with the Seastar I/O Scheduler 2.0, which helps balance I/O workloads more effectively across shards. A new API for removing nodes from gossip and several changes to reduce latency spikes are introduced, alongside updates to the Change Data Capture (CDC) API. ScyllaDB 4.4 also sees additions in tools and diagnostics, as well as the transition from using gcc to clang for building, supporting coroutines necessary for Raft implementation. The update allows for hinted handoff configuration changes without restart and adds monitoring enhancements, such as log collection support via Grafana Loki, improved error metrics, and several bug fixes to enhance stability and performance.

In the aftermath of a fire at the OVHcloud Strasbourg datacenter, Kiwi.com demonstrated remarkable resilience due to its use of ScyllaDB, a NoSQL database known for high availability and disaster recovery capabilities. The fire, which began on March 10, 2021, and resulted in the destruction of several data rooms, affected millions of websites. Despite losing ten out of thirty server nodes, Kiwi.com's ScyllaDB cluster continued to operate effectively by leveraging its multi-local infrastructure, designed to distribute data across three geographically separated datacenters. This setup allowed for seamless load redistribution and zero data loss. The incident highlighted the importance of disaster planning and testing full datacenter outages regularly, as well as maintaining sufficient capacity to handle unexpected events. Kiwi.com's foresight in creating a robust system ensured minimal service disruption, underscoring the value of investing in geographically distributed data replication.

The announcement introduces a shard-aware C/C++ driver for ScyllaDB, an API-compatible fork of the Datastax C++ driver, designed to optimize performance through shard-awareness, allowing client applications to query specific CPUs within a ScyllaDB cluster. This driver, compatible with Apache Cassandra and DataStax Enterprise, enhances query efficiency by binding each CQL connection to a specific CPU shard. The driver supports both basic and advanced modes of shard selection, with the advanced mode using a dedicated Scylla-specific port for optimal shard targeting. Installation instructions are provided for CentOS 7 and Ubuntu 18.04, while the driver can also be compiled from source for other Linux distributions. This development marks a significant performance enhancement for ScyllaDB users, with the new driver offering automatic improvements without requiring changes to existing client code.

Dan Podhola, a Principal Software Engineer at Zillow, presented at the ScyllaDB Summit on managing optimistic concurrency with write-time timestamps to address the challenge of processing out-of-order messages in their high-throughput backend systems. Zillow's system handles property and listing data, translating it into a common format for internal communication, while ensuring the most current information is always processed. The team encountered issues with out-of-order data due to concurrent message queues, which could lead to inaccuracies and require manual intervention. To resolve this, they implemented a method using ScyllaDB's "USING TIMESTAMP" feature, which allows them to ensure only the newest data is written without relying on traditional SQL locking or Lightweight Transactions. This strategy enhances scalability and performance, allowing Zillow to process over 6,500 records per second without impacting real-time workloads, ensuring efficient data handling across their systems.

QOMPLX, a company focused on cybersecurity, has leveraged the capabilities of graph databases and partnered with Expero to implement JanusGraph with ScyllaDB as a storage layer to address evolving cybersecurity threats. By ingesting data from Microsoft Active Directory into graph representations, QOMPLX's Q:CYBER platform enhances the ability of cybersecurity analysts to detect potential attack paths and vulnerabilities in networks. The platform's graph canvas allows for interactive exploration of network data, aiding analysts in understanding potential security breaches and the "blast radius" of vulnerabilities. The architecture involves a collector service for data ingestion, using various data stores like ScyllaDB, Apache Spark, and Elasticsearch to accommodate different analytics needs. Expero's Brian Hall highlighted challenges in graph analytics, such as computational expense and managing multiple graphs, which led to developments like BoostGraph for specific computations and revisions to JanusGraph. The integration of monitoring tools like Prometheus and Grafana ensures data consistency across platforms, with ScyllaDB noted for its reliability in this multi-technology setup.

Change Data Capture (CDC) in ScyllaDB, which became production-ready in version 4.3, allows users to track and respond to data changes via a CQL-compatible interface, enabling existing tools or drivers to process CDC data. The implementation of CDC involves creating a "CDC log" table for each CDC-enabled table, where changes in the base table are logged in streams corresponding to portions of the token ring. Initially, stream IDs were generated using a computationally expensive Las Vegas-type algorithm, but a new deterministic approach now simplifies this process by directly encoding tokens into stream IDs. However, this new method can confuse shard-aware drivers that still use MurmurHash3, potentially leading to increased latency when queries are sent to incorrect nodes or shards. The issue is addressed by teaching drivers to detect custom partitioning schemes and correctly compute tokens for partition keys. ScyllaDB is updating its drivers to support this feature, with GoCQL and Java drivers already updated, and plans for C++, Python, and Rust drivers underway.

Benchmarking ScyllaDB requires careful consideration of numerous factors and methodologies to ensure accurate, fair, and reproducible results. This involves understanding the intricacies of system load generation, latency measurement, and the impact of concurrency on performance testing infrastructure. Effective benchmarking also requires addressing coordinated omission, utilizing probability distributions, and applying statistical and queuing theories. Developers need to account for specific workload patterns such as read-heavy or write-heavy operations, and should carefully configure database and system layers to reflect real-world use cases. Additionally, choosing appropriate open-source frameworks like YCSB, and understanding the unique architecture of ScyllaDB, such as its thread-per-core design, are crucial for performance optimization. The goal is to achieve desired throughput and latency targets, such as maintaining a 99th percentile latency within acceptable limits, while ensuring that the system is not overloaded. The article emphasizes the importance of isolating and monitoring the performance of both the target system and the data loaders, and suggests using ScyllaDB's shard-aware drivers for optimal results.

Project Circe is an initiative aimed at enhancing ScyllaDB's capabilities in consistency, performance, scalability, stability, manageability, and ease of use. A significant focus of the February 2021 update is the integration of the Raft consensus protocol, which introduces leader-based log replication to ScyllaDB, differing from the previously used leaderless Paxos protocol. Raft's implementation facilitates topology changes, ensuring consistent configuration updates across nodes, and addresses the challenges posed by the slower gossip protocol. It also supports linearizable schema changes, resolving issues with eventual consistency. Additionally, Raft enables the introduction of "tablets," a concept borrowed from Google Bigtable, for dynamic load balancing and efficient data distribution. Recent developments include Raft-related commits and improvements to the DynamoDB-compatible interface, Alternator, alongside progress in ScyllaDB's Kubernetes operator.