January 2019 Summaries

ScyllaDB Monitoring Stack 2.1, released by the ScyllaDB team, is an open-source monitoring solution for ScyllaDB Enterprise and Open Source, leveraging Prometheus and Grafana. This version supports various ScyllaDB Open Source and Enterprise versions, as well as ScyllaDB Manager 1.3. Key updates in this release include an upgrade to Grafana 5.4.2, which introduces significant layout changes and a new URL path structure for dashboards. The monitoring stack now employs local file configurations for dashboards, plugins, and data sources, enhancing safety and upload speed. Additionally, clickable alerts allow users to navigate directly from alerts to specific dashboard views for detailed analysis. Customized dashboards are now more flexible due to backward compatibility with Grafana 5.x, although some attributes like row height are no longer supported. The make_dashboard.py script has been updated to generate Grafana 5 format dashboards with improvements in layout positioning.

ScyllaDB Open Source 3.0 introduces a new on-disk file format, SSTable 3.0, which offers significant improvements in data storage and retrieval efficiency, particularly in terms of disk space savings and compatibility with Apache Cassandra 3.x. The new format, referred to as "mc," addresses several limitations of the older formats, such as data duplication and inefficient data structure alignment with CQL, by introducing row-based storage and separating metadata from the data files. This restructuring allows for more efficient binary searching and reduces the need for data compression, potentially enhancing performance. With examples showing disk space savings ranging from a few percentage points to over 50%, the new format is especially beneficial for schemas with wide rows and long column names, though the actual advantages can vary based on specific use cases. While SSTable 3.0 is not enabled by default in version 3.0 to ensure a smooth transition, future releases plan to make it standard, alongside further performance enhancements.

ScyllaDB has announced new maintenance releases for its open-source database software, with versions 3.0.1 and 2.3.2 now available. Both releases are backward compatible and support rolling upgrades, ensuring ease of transition for users. Version 3.0.1 addresses issues such as startup problems on large machines and the application of limits in filtering features, while version 2.3.2 fixes bugs related to incorrect bandwidth reporting, Linux kernel upgrade compatibility, and memory allocation failures. Users are encouraged to upgrade to the latest stable release, ScyllaDB 3.0, to benefit from these improvements and additional features.

ScyllaDB Open Source 3.0 introduces significant enhancements in performance and functionality for the NoSQL database, with key improvements in streaming and hinted handoff processes. The new release optimizes the data streaming process between nodes by replacing the previous RPC-based method with a specialized interface that reduces overhead, resulting in faster operations such as node additions and decommissioning. These changes are demonstrated to improve streaming performance by up to 22% over the previous version, ScyllaDB 2.3. Additionally, the introduction of the Hinted Handoff feature reduces data discrepancies during node outages, decreasing the need for on-the-spot data reconciliation during QUORUM consistency reads and improving tail latencies by up to 62%. These advancements, alongside the transition of features like Materialized Views and Secondary Indexes to production-ready status, collectively enhance the capabilities and efficiency of ScyllaDB, offering users improved data modeling and workload handling.

Grab, a prominent mobile app in Southeast Asia, integrates various services like mobile payments, ridesharing, food, and package delivery, similar to a combination of Apple Pay, Lyft, and DoorDash. Originating in Malaysia and Singapore, it has expanded across several Southeast Asian countries, including Cambodia, Vietnam, Thailand, Myanmar, Indonesia, and the Philippines. At the ScyllaDB Summit, Grab's Engineering Lead, Aravind Srinivasan, detailed the company's use of ScyllaDB as a persistent database for storing data from Apache Kafka, a stream processing platform, and how they employed ScyllaDB for their fraud detection system. The presentation emphasized Grab's microservices architecture developed in Golang and highlighted ScyllaDB's role in their technological infrastructure.

ScyllaDB Open Source 3.0 is a significant update to the open-source NoSQL database designed to enhance performance and efficiency, offering features that surpass Apache Cassandra. Built from scratch in C++, ScyllaDB provides improved database performance with 10 times the throughput and consistent low latency, requiring fewer nodes due to its close-to-the-hardware design. The update introduces several key features, including Materialized Views, which automate and simplify data organization, and global secondary indexes, which offer scalable indexing independent of cluster size. The new version also supports a more efficient storage format, reducing storage volume by up to 66%, and enhances streaming and paging processes to improve system responsiveness and data handling. Hinted handoffs are now production-ready, improving data consistency during temporary node outages. Additionally, the update includes server-side filtering to enhance query flexibility and compatibility with Apache Cassandra, offering benefits such as reduced storage needs and improved CPU utilization.

ScyllaDB announced the release of ScyllaDB Enterprise 2018.1.9, a minor update aimed at addressing specific bugs in the 2018.1 branch of their NoSQL database. This update fixes two key issues: an assertion failure that occurred during certain queries involving counter tables without live rows, and incorrect latency values reported by Prometheus. ScyllaDB encourages its Enterprise customers to upgrade to this version with assistance from their support team. The release is available for current customers or as a 30-day evaluation, with upgrade paths provided from various previous versions.

ScyllaDB, an open-source NoSQL database, began supporting the JSON format starting with version 2.3, allowing users to insert and retrieve JSON documents and transform between native CQL types and JSON without client-side conversions. While schemas are still enforced, the JSON support in ScyllaDB facilitates easier manipulation of data through CQL statements like SELECT and INSERT JSON, enabling users to define and query structured data conveniently. The JSON format allows for intuitive data handling, with omitted columns defaulting to null or utilizing the DEFAULT UNSET flag to maintain existing values. Functions like fromJson() and toJson() further enhance data manipulation by converting single values to and from JSON format, supporting complex data structures such as collections. This feature offers opportunities for innovative database designs and implementations, encouraging developers to explore JSON's capabilities within ScyllaDB environments.

ScyllaDB Summit 2018 featured over three dozen sessions, including keynotes, breakout tracks, and tech talks, which are now accessible online through videos and slides. These sessions covered various topics such as use cases, engineering insights, and upcoming features, with presentations from ScyllaDB executives, prominent customers, and community members. Notably, ScyllaDB CTO and Co-Founder Avi Kivity discussed both near-term and longer-term initiatives, coinciding with the upcoming release of ScyllaDB Open Source 3.0. The materials are available on YouTube and SlideShare, and the organization encourages viewers to engage with them by asking questions or sharing their experiences with ScyllaDB.

ScyllaDB Enterprise 2018.1.8, a minor release announced by the ScyllaDB team, introduces important bug fixes and enhancements to the latest stable branch of the enterprise NoSQL database. This release incorporates memory management improvements from the ScyllaDB Open Source 2.3 release, which enhance the database's ability to more reliably free large contiguous memory areas, thereby improving performance for workloads with large blobs or collections. Customers are advised to upgrade in coordination with ScyllaDB support to benefit from these enhancements and other issue resolutions, such as preventing segmentation faults during extreme out-of-memory situations and adjusting audit configuration syntax in the scylla.yaml file.