May 2021 Summaries

Avishai Ish Shalom from the ScyllaDB team discusses the ease of scaling databases to handle large workloads using ScyllaDB Cloud's pricing and sizing calculator, which aids in determining the appropriate cluster size based on peak workload requirements. The article details a test conducted with ScyllaDB Cloud to manage a sustained workload of 1 million operations per second, revealing that the recommended cluster, although more robust than necessary, can handle nearly 2 million operations per second with low latency. The test highlights the importance of benchmarking potential configurations before settling on a cluster size, as real-world performance can vary due to factors like data accumulation and compactions. The process of setting up and running load tests in the cloud is simplified through automated provisioning and pre-configured templates, making it accessible for users to validate the performance of their ScyllaDB clusters.

ScyllaDB Enterprise 2021 marks a significant advancement for the NoSQL database, positioning itself as a drop-in replacement for both Apache Cassandra and Amazon DynamoDB, while offering unique enterprise-only features and enhanced reliability. This release introduces new capabilities such as the Space Amplification Goal (SAG) for fine-tuning between space and write amplification, and improvements in the Gossip protocol by removing the seed concept. Additionally, it offers a binary search in SSTable Promoted Index, which drastically enhances index lookup performance, and the Alternator API enhancements, which facilitate smoother integration for DynamoDB users with features like load balancing and SSL options. The Unified Installer simplifies deployment on servers without root access, while ongoing security updates ensure enhanced protection through features like hot reloading of SSL certificates and shard-aware ports. ScyllaDB Enterprise 2021 is available immediately for existing customers, with a 30-day trial for new users, and includes comprehensive documentation and support for those looking to explore its capabilities.

Pekka Enberg, a key figure in the development of ScyllaDB, discusses the challenges and evolution of kernel APIs and architecture in relation to modern hardware. His work with Seastar, a framework that bypasses traditional OS abstractions, highlights the inefficiencies of legacy systems like POSIX, which were designed for single-CPU models. The Seastar framework aims to improve performance by eliminating context switches and synchronization issues, offering control over machine resources through a non-blocking, future/promise model. Enberg argues that while traditional abstractions like POSIX have historically provided portability across systems, modern hardware advancements, particularly in networking and storage, challenge the relevance of such models. He suggests that the costs of maintaining these outdated interfaces are becoming prohibitive, especially as CPU speeds plateau and storage technologies advance. Enberg's insights suggest a potential shift towards new abstractions that better align with current hardware capabilities, emphasizing efficiency and control in software design.

Helm Charts have become a popular tool for deploying applications like ScyllaDB on Kubernetes due to their ability to simplify and customize deployments by packaging Kubernetes manifests into a single, easily installed package. With the introduction of ScyllaDB Operator 1.1, users can now utilize three specific Helm Charts—scylla-operator, scylla-manager, and scylla—to streamline the deployment and management of ScyllaDB products. These charts are available in two channels: the "latest" for the newest builds and the "stable" for QA-approved releases, with the latter recommended for production environments. Users have the flexibility to override default parameters during installation to meet specific requirements, facilitating quick and efficient deployments. The process involves installing the ScyllaDB Operator and necessary dependencies, such as Cert Manager, and customizing deployments for different configurations, such as multi-node clusters across various racks. The Helm Chart source code is accessible in the ScyllaDB Operator GitHub repository, providing further customization options and support for users seeking to enhance their ScyllaDB deployments.

Mapped has introduced a versatile AI-powered platform that consolidates various data sets from the Industrial Internet of Things (IIoT) into a unified system, enabling businesses to monitor and manage their facilities efficiently. The platform uses a GraphQL API and industry-standard ontologies to support the integration of over 30,000 device models across 900 classes, facilitating the development of tailored applications for diverse business needs. By employing JanusGraph for graph data modeling and ScyllaDB for its robust NoSQL database capabilities, Mapped ensures scalability and high performance, which is crucial for maintaining stringent "five nines" service level agreements. The company primarily targets tenants in commercial real estate but plans to expand into sectors like manufacturing, healthcare, and logistics. Mapped's platform leverages machine learning to uncover data relationships, allowing for predictive analytics and enhanced operational insights.

ScyllaDB Manager 2.3 introduces a feature that allows users to suspend and resume operations, facilitating maintenance windows or off-peak strategies by preventing tasks like repairs from running during prime time. When a cluster is suspended, only health check tasks are permitted, and no new tasks can be scheduled or manually run within the next eight hours. Users can suspend a cluster by executing the "sctool suspend" command and verify the suspension using the "sctool task list" command, where suspended tasks are marked with a [SUSPENDED] prefix. Resuming a cluster does not automatically restart stopped tasks unless the "--start-tasks" flag is used; tasks instead follow the "Next run" schedule. Automation of this process during peak hours can be achieved using crontab or an external scheduler via the ScyllaDB Manager API. The tool is available for free for ScyllaDB Open Source users with up to five nodes and offers unlimited use for ScyllaDB Enterprise users.

Project Circe is a year-long initiative by ScyllaDB aimed at enhancing the capabilities of the NoSQL database. In April, the focus was on integrating the Raft consensus protocol, which involves considerable code development and rigorous testing to ensure resilience and functionality. This includes over a hundred test cases and the use of a logical clock to simulate millions of events per second. The project has also seen improvements in compaction strategies and memory allocation to reduce latencies and CPU stalls, particularly when dealing with large data blobs. Additionally, ScyllaDB Monitoring Stack 3.7 has been released, offering enhanced visibility of Hinted Handoffs during transient node failures. For those new to ScyllaDB or NoSQL databases, ScyllaDB University provides free online resources to build database skills, ranging from basic overviews to advanced concepts like consensus protocols.