November 2016 Summaries

ScyllaDB 1.4.2, a bugfix release of the ScyllaDB 1.4 stable branch, has been announced, addressing several issues while maintaining backward compatibility and supporting rolling upgrades. The release fixes a range of bugs, including those affecting internal queries to the authentication table, startup issues on machines with many cores, uninformative error messages, and handling of certain CQL and Thrift commands. It also resolves problems with Nodetool operations, JMX metrics, and cache behavior during range queries. Contributions from various developers have enhanced the system's stability by correcting histogram and moving average calculations, improving exception handling, and refining sstable management, ensuring smoother and more reliable database operations.

ScyllaDB 1.5 RC1, an open-source NoSQL database compatible with Apache Cassandra, has been released, emphasizing stability, bug fixes, and workload conditioning. This release introduces the capability for ScyllaDB to self-tune by automatically reducing request rates when disk speeds are insufficient, ensuring optimal performance. Notable bug fixes address issues like unnecessary parallel queries during range scans, file descriptor retention causing disk space issues, negative cache capacity values, and memory leaks in snapshot operations. New metrics have been added, such as counters for replica writes delayed by excessive unflushed memory and blocked requests in the commitlog write path due to slow disk speeds, while some older metrics have been rendered obsolete by these workload conditioning improvements. The release is not yet production-ready, and further testing is encouraged to help stabilize the version by reporting any encountered bugs.

As of ScyllaDB version 1.4, native support for Prometheus facilitates system monitoring by allowing seamless metrics exporting and integration with visualization tools like Grafana. ScyllaDB gathers various performance metrics, such as memory consumption and cache operations, which are crucial for maintaining system health and are utilized in tools like the nodetool utility. Prometheus, a project under the Cloud Native Computing Foundation, serves as a robust platform for monitoring, offering capabilities like graph visualization, mathematical computation, and alert triggering based on metric evaluations. The integration with ScyllaDB requires minimal configuration and leverages the same infrastructure as the previous collectd API, making it straightforward to add ScyllaDB servers to Prometheus targets. Additionally, ScyllaDB's setup script facilitates the installation of node_exporter for machine metrics collection, enhancing the comprehensiveness of the monitoring solution. Future enhancements to the ScyllaDB Prometheus API include the addition of histogram support and detailed metrics descriptions to improve monitoring efficacy.

ScyllaDB's blog post by Glauber Costa delves into the challenges of diagnosing and resolving unexpected high latencies in the database's performance, attributed to a bug in the Linux kernel's CFQ I/O elevator. When ScyllaDB users experienced unpredictable write timeouts, Costa and the team embarked on a thorough investigation using tools like systemtap and eBPF to track down the source of the latency spikes. They discovered that flush operations were being delayed due to the serialization of blocking syscalls, with file opening operations taking an unusually long time because of locks within the XFS filesystem. The root cause was identified as a starvation problem within the CFQ scheduler, which deferred requests for an unacceptably long duration, up to 15 seconds. The team addressed the issue by switching to the noop I/O elevator in ScyllaDB's 1.4 release and contributing a patch to the Linux kernel to resolve the bug. This fix was subsequently integrated into upstream Linux and backported to stable kernel versions, providing a more stable experience for ScyllaDB users.

The blog post provides a concise introduction to deploying and managing ScyllaDB nodes using Docker, detailing the process of starting a single node, accessing utilities like nodetool and cqlsh, and forming a cluster by joining additional nodes. It emphasizes the importance of configuring Docker for regular user access, explaining how to start a ScyllaDB instance, monitor server logs, and handle cluster status with nodetool. The post also discusses improving storage performance through Docker volumes and configuring resource limits to optimize ScyllaDB's use of system resources. It highlights the necessity of using appropriate command line options for efficient operation in overprovisioned environments, ensuring that ScyllaDB runs smoothly within Docker containers.

ScyllaDB 1.4.1, released on November 8, 2016, is a bugfix update for the ScyllaDB 1.4 stable branch, ensuring backward compatibility and support for rolling upgrades. The primary fix in this release addresses an issue where selecting the same column twice would trigger an assertion error, as noted in bug #1367. The release reflects ScyllaDB's ongoing commitment to maintaining stability and addressing bugs in their database management system.

ScyllaDB 1.3.4, released on November 7, 2016, is a bugfix update for the ScyllaDB 1.3 stable branch, ensuring backward compatibility and supporting rolling upgrades. Users are encouraged to upgrade to the latest stable ScyllaDB 1.4 for improved features. This release addresses specific issues, such as an assertion error caused by selecting the same column twice and a failure to restart scylla-jmx after the Scylla server is killed. Contributions from Pekka Enberg include fixes for these bugs, and updates are provided for both the main ScyllaDB and its JMX components. The release highlights the importance of maintaining system stability and encourages users to report problems and contribute to ongoing development.

ScyllaDB 1.4, a minor release of the open-source NoSQL database compatible with Apache Cassandra, introduces several enhancements aimed at improving performance and monitoring. This version includes a new Slow Query Tracing feature that helps identify latency issues with minimal overhead, and it offers native support for Prometheus metric collection, eliminating the need for the collectd intermediary. Additionally, ScyllaDB 1.4 provides new metrics for streaming, database operations, gossip, tracing, and cache, and it supports Snappy compression for CQL binary protocol. The release also addresses various bugs, enhancing repair speed, reliability, and data streaming, while introducing a legacy Random Partitioner for compatibility with Cassandra users. Significant improvements have been made to the repair process and bug fixes have been implemented to prevent crashes, improve performance, and ensure stability across different configurations and use cases.