September 2018 Summaries

ScyllaDB 2.3 introduces significant performance improvements over its predecessor, ScyllaDB 2.2, by focusing on three main areas: enhanced CPU scheduling, reduction of latency-inducing events, and a newly designed I/O Scheduler aimed at delivering lower latencies during peak throughput. The update demonstrates improved tail latencies for CPU-intensive workloads, with read latency improvements of up to 25% in higher percentiles, and write latency enhancements of up to 11% in similar scenarios. Additionally, the update offers substantial gains in I/O-bound read latencies during major compactions, achieving up to 85% better performance in certain percentile measurements. These advancements are largely attributed to the improved I/O Scheduler's ability to maintain performance during background operations and the refined process of CPU scheduling. ScyllaDB 2.3 also simplifies deployments by addressing inconsistencies in I/O configuration, making it a more reliable and efficient option for handling both disk-bound and CPU-intensive workloads.

ScyllaDB Monitoring Stack 2.0, announced by the ScyllaDB team, is an open-source monitoring solution for both ScyllaDB Enterprise and Open Source, utilizing Prometheus and Grafana for enhanced dashboard usability and improved underlying technology. The update introduces support for ScyllaDB Open Source versions 2.1 to 2.3, Enterprise versions 2017.x and 2018.x, and ScyllaDB Manager 1.1, with a notable shift to Prometheus version 2.3.2, which offers significant improvements in storage format but lacks backward compatibility with previous versions. This release includes features like multi-cluster and multi-data center dashboards, where users can filter charts by cluster or data center, and the identification of nodes by IP addresses instead of hostnames to unify metrics. The upgrade process, which may be complex due to the compatibility changes, is supported by guides and community resources, with enterprise users encouraged to seek proactive assistance from ScyllaDB.

ScyllaDB 2.3, a minor release of the open-source, Apache Cassandra-compatible NoSQL database, introduces several enhancements and experimental features aimed at improving performance and functionality. These include improvements in CQL with support for JSON operations and datetime functions, performance optimizations through improved handling of large partitions, and new tooling such as ScyllaDBtop and iotune v2. The release also offers new distribution support for Ubuntu 18.04 and Debian 9, and experimental features like Materialized Views, Secondary Indexes, and Hinted Handoff. The update emphasizes reduced latency, increased throughput, and better performance isolation, while maintaining compatibility with existing Cassandra formats, with plans for further enhancements in future releases.

The Distributed Data Summit 2018 in San Francisco brought together NoSQL practitioners with a focus on Apache Cassandra and its ecosystem. Keynote speaker Nate McCall highlighted the current state and updates of Cassandra 4.0, including features like Transient Replicas and Zero Copy Streaming, and addressed issues such as the legacy "tick-tock" release model and the state of Materialized Views. The summit also saw discussions on the divergence between commercial vendors and open-source contributors. A session by ScyllaDB CEO Dor Laor examined the decision to base ScyllaDB on Cassandra, focusing on the benefits of inherited features alongside operational challenges. ScyllaDB's CTO Avi Kivity presented on the advantages of ScyllaDB's thread-per-core architecture, showcasing its potential for reducing hardware needs and improving performance. Additionally, Kivity explored how ScyllaDB can support both OLTP and OLAP workloads on the same infrastructure through workload isolation, with promising results. The event underscored ongoing innovations and challenges in the NoSQL space, with a focus on improving performance, scalability, and operational management.

ScyllaDB 2.3 introduces enhanced support for managing large partitions, which are known to cause performance issues, by utilizing the system.large_partitions table to identify and analyze them. The update aims to improve data modeling and performance by allowing users to track large partitions within a cluster, using a schema that records key attributes like keyspace name, table name, partition size, and compaction time. The table functions as a node-local repository and updates its entries during memtable flushes and compaction, providing insights into the distribution of large partitions. Users can configure the threshold for what constitutes a large partition through the compaction_large_partition_warning_threshold_mb parameter in the scylla.yaml file, with the default set to 100MB. Although the current release does not support the ALLOW FILTERING keyword in queries, this feature is planned for future updates. The introduction of system.large_partitions reflects ScyllaDB's ongoing commitment to improving large partition handling, encouraging users to explore and provide feedback on their experiences with managing data in this context.

In a seminar, Miguel Martinez Pedreira from CERN and Glauber Costa from ScyllaDB discussed the use of ScyllaDB for real-time processing of big data in the ALICE experiment at CERN, focusing on its application to the AliEn Global File Catalogue. This catalogue, a comprehensive metadata index of files from the experiment, spans 80 computer centers across five continents and requires efficient handling of real-time data. ScyllaDB, known for its close-to-the-hardware architecture, enabled CERN to achieve high throughput and low latency while maintaining tight control over total cost of ownership (TCO). The transition from Cassandra to ScyllaDB in this context demonstrated significant improvements in processing capabilities, as highlighted in their video presentation.