May 2025 Summaries

ScyllaDB has transitioned from mutation-based streaming to file-based streaming, resulting in a 25-fold increase in streaming speed and a 10-time improvement in network bandwidth. This shift involves streaming entire SSTable files directly between nodes without the need for deserialization and re-serialization, reducing CPU usage significantly, especially for data models with small cells. The compact nature of SSTable files compared to mutation fragments reduces the amount of data transmitted over the network, enhancing efficiency and performance. Tests conducted on ScyllaDB nodes demonstrated much lower CPU usage and faster data transfer with the new method, marking a significant improvement in ScyllaDB's data streaming capabilities. This new streaming approach is available in ScyllaDB Cloud and the ScyllaDB 2025.1 release.

ReversingLabs successfully executed a massive migration of over 300 terabytes of data and 400 services from their internally-developed key-value database to ScyllaDB without any downtime. Initially limited by database options when they started in 2011, the company built their own efficient key-value store to handle their unique demands as a startup. However, as their system complexity grew, they opted for ScyllaDB due to its speed, resilience, and scalability. The migration strategy involved maintaining the existing database interface, making minimal changes to the data model, and leveraging Kafka for streaming data. The service migration was carefully staged, starting with APIs, followed by master and delivery services, ensuring continuity for API users. A significant challenge was adapting to ScyllaDB’s leaderless architecture, which necessitated developing a new distributed locking mechanism. The transition was seamless, resulting in improved system resiliency and high availability.

ScyllaDB has introduced a new tablet-based data distribution system that significantly enhances the performance of full table scans, which are essential for analytical queries. Unlike the previous vNode-based approach, tablets improve scan efficiency by eliminating complex tuning heuristics and allowing more precise targeting of data across replicas and shards. Tablets create a dynamic, on-demand structure that adapts to varying workload requirements, thereby addressing the inefficiencies of static vNode topologies such as uneven load distribution and resource contention. This new system provides deterministic scanning by setting a geometric average size for tablets, which can be configured to suit specific needs, and it streamlines processing by reducing cross-CPU and node communication. ScyllaDB’s tablets also integrate with existing systems, offering backward compatibility with vNodes while enabling full scans to run more efficiently, benefiting from workload prioritization and cache bypassing to maintain low latency during concurrent operations.

ScyllaDB, a distributed NoSQL database known for its speed and scalability, has evolved significantly over the past decade, transitioning from a focus on raw performance to embracing cloud capabilities and advanced features. Initially positioned as a faster alternative to Cassandra, ScyllaDB has expanded its functionality to include materialized views, secondary indexes, and third-party integrations, achieving feature parity with Cassandra. The introduction of ScyllaDB Cloud in 2019 marked a pivotal shift, with the platform growing rapidly and attracting major clients like Discord and Tripadvisor. Recent advancements include the adoption of the Raft protocol and the introduction of tablets, facilitating faster and more efficient scaling by enabling parallel node updates and consistent data distribution. These innovations allow ScyllaDB to achieve higher storage utilization rates, translating to cost savings for users. Additionally, ScyllaDB is enhancing its database with features like S3 object storage support, workload prioritization, and vector capabilities, catering to diverse workloads and AI applications. Through continuous improvements and a focus on core database principles, ScyllaDB remains committed to delivering high performance and adaptability to its users.