May 2023 Summaries

The blog post introduces expanded tutorials for developing applications using ScyllaDB, specifically focusing on an IoT project named CarePet designed to monitor pets' health through virtual collars. The tutorials now include examples in Rust, Python, and PHP, alongside a Terraform example for setting up a ScyllaDB Cloud cluster. The CarePet app comprises three main components: a virtual collar that gathers sensor data, a web app for data analysis, and a database migration tool. The project is intended as a learning tool rather than a production-ready solution, with a focus on scaling to handle data from 10 million pets, translating to 43 billion data points per month. The tutorials guide users through the development process, including data modeling and the necessary infrastructure, while emphasizing ScyllaDB's performance capabilities and cost benefits compared to other databases like DynamoDB. The post also encourages engagement with the ScyllaDB community for support and further learning through various resources.

High concurrency is crucial for optimal database performance, but overwhelming the database with excessive requests can lead to decreased throughput and increased latency. In a recent webinar, Brian Taylor from Optimizely discussed how to maximize database concurrency effectively by understanding the Universal Scaling Law (USL), which describes how system throughput changes with varying user numbers. The USL identifies three regions: linear, saturation, and retrograde, with the linear region being where throughput is directly proportional to concurrency, while the saturation region sees constant throughput, and the retrograde region experiences decreasing throughput with increased concurrency. To address these challenges, adaptive concurrency limits and techniques like the TCP congestion control algorithm are suggested for dynamically managing concurrency. Additionally, factors such as payload size and the nature of workloads, whether spiky or consistent, impact concurrency and require careful tuning to maintain desired throughput. Taylor also developed a tool for open-loop testing with ScyllaDB, which complements existing closed-loop testing methods by simulating a constant throughput load, providing insights into real-life system behavior under varying conditions.

Tencent Games developed a real-time event-driven analytics system using ScyllaDB and Apache Pulsar, leveraging Command and Query Responsibility Segregation (CQRS) and event sourcing patterns to enhance efficiency in monitoring gameplay activities. This architecture enables the seamless handling of large-scale data events by separating data models for reads and writes, allowing independent scaling and reducing conflicts across different teams. ScyllaDB plays a crucial role in efficiently dispatching events to numerous gameplay sessions by using session IDs as partition keys and event IDs as clustering keys, addressing the challenges of event consistency and data distribution. The system simplifies global data management through data replication across keyspaces, facilitating compliance with data protection laws and eliminating the complexities of implementing distributed systems. By utilizing keyspaces as data containers, Tencent ensures transparency and improves the auditability of data distribution while maintaining scalability and efficiency. This innovative approach allows the company to address risks such as cheating activities and harmful content, ensuring a seamless gaming experience for millions of players worldwide.

ScyllaDB is advancing its path to strong consistency by introducing Raft-based consistent and centralized topology management in its Open Source 5.4 release, aiming to address limitations like the inability to perform more than one topology operation at a time and the risk of lost writes during topology changes. The new approach eliminates the dependence on ring delay and introduces a centralized topology change coordinator, which drives all topology changes alongside the Raft cluster leader, ensuring that state changes are strongly consistent and can be executed safely and quickly. This implementation includes a new feature called "fencing," which ensures that reads and writes are signed with the current topology version to prevent outdated or incompatible operations, thus resolving consistency anomalies and enhancing the reliability of topology changes. Additionally, the system now supports automatic coordinator failover, ensuring that topology changes can progress even if the coordinator fails, as a new coordinator can take over seamlessly. The changes aim to make topology operations more efficient, reliable, and less prone to operator errors, while the ScyllaDB community looks forward to enhanced scalability and fault tolerance.

Cynthia Dunlop's blog post from May 2023 discusses the evolving landscape of NoSQL databases, highlighting trends and tradeoffs associated with their adoption, especially in contrast to SQL and NewSQL databases. The discussion, featuring insights from ScyllaDB's Tzach Livyatan and TDWI's James Kobielus, emphasizes the increasing demand for wide column NoSQL databases like Cassandra and Bigtable due to their ability to handle massive amounts of data generated by global, low-latency applications in sectors such as gaming, IoT, and media streaming. The tradeoffs of NoSQL include variations in data distribution, transaction support, and latency, which differ across the many types of NoSQL databases. The article also explores how NewSQL databases attempt to merge the benefits of both SQL and NoSQL, albeit with some latency costs, and considers the future role of Kubernetes in database orchestration, noting the challenges and potential solutions for maintaining performance in a cloud-native environment.

ScyllaDB's upgrade to version 5.2 marks a significant shift towards strong consistency by employing the Raft consensus algorithm, particularly for schema management, enhancing the reliability and speed of schema changes. Previously relying on eventual consistency, ScyllaDB now uses Raft to ensure that schema changes are propagated quickly and without conflict, requiring a majority of nodes to acknowledge changes, thus increasing stability during concurrent operations. The upgrade introduces features like consistent cluster management, disaster recovery options, and support for IP address changes, facilitating seamless operations in environments like Kubernetes. Although Raft is taking over several functionalities, the Gossip protocol remains integral for tasks such as node liveness detection, albeit with reduced responsibilities. This transition aims to make ScyllaDB more robust and efficient, while maintaining compatibility with existing tools and operations.

ScyllaDB Open Source 5.2, building on the 2022 major release, introduces significant updates including Raft-based strongly consistent schema management, which enhances performance, stability, and ease of use by serializing schema management operations via the Raft consensus algorithm. This release addresses over 100 issues and empowers users to perform concurrent schema changes safely, with schema propagation occurring more swiftly. The update also sees the promotion of Time To Live (TTL) for the ScyllaDB's DynamoDB API (Alternator) to production readiness, allowing configurable expiration delays. Enhancements include the detection of large collections to avoid performance degradation, automated management of tombstone garbage collection, and improvements to prevent timeouts when processing extensive tombstone sequences. Additionally, secondary indexes can now be applied to collection columns, enhancing data retrieval capabilities. These updates, alongside various other improvements in performance, stability, and operational efficiency, make ScyllaDB 5.2 a robust choice for NoSQL database solutions.