June 2023 Summaries

ScyllaDB, in collaboration with the University of Warsaw, has developed a new Rust library for consuming Change Data Capture (CDC) logs, which is faster and uses less CPU than its Go and Java counterparts. This initiative involved a group of four students who, under the guidance of ScyllaDB engineers, created a library that enables real-time data processing and integration with other systems, essential for immediate reactions to database modifications. The library, written in pure Rust using the Scylla Rust Driver and Tokio, features a callback-based interface, automatic error retries, and optional checkpointing for progress saving. Benchmarks indicate that the Rust library outperforms the Java driver and is at least as fast as the Go library while consuming significantly less CPU. This project, forming the basis of the students' Bachelor's thesis, demonstrates Rust's potential in building efficient applications connected to ScyllaDB clusters, highlighting its user-friendly interface through example applications and benchmarks. The library's development, alongside ScyllaDB's ongoing support for Rust, underscores a shift toward more efficient and scalable solutions in data processing.

In the blog post by Michał Jadwiszczak, the advantages of distributing aggregation calculations in distributed databases, specifically ScyllaDB, are explored to significantly enhance query execution efficiency. Traditional monolithic databases struggle with complex queries in large datasets due to their centralized approach, leading to potential performance bottlenecks. By distributing the workload across multiple nodes, each node processes its local data and returns partial results, which are then combined by a super-coordinator node, resulting in up to 20 times faster query execution. The post differentiates between native aggregates and user-defined aggregates (UDAs), highlighting the flexibility of UDAs in executing tailored calculations through user-defined functions. Despite the enhanced performance, limitations exist, such as the inability to distribute queries with filtering or grouping clauses without further technical implementations. Benchmark tests on AWS instances demonstrate the notable improvements in execution time and reduced network traffic, underscoring the transformative potential of distributed aggregates in modern data processing environments.

Numberly has utilized both ScyllaDB and MongoDB in production for over five years, strategically deploying each based on specific use cases and technical needs. While MongoDB's ease of adoption and flexibility make it ideal for web backends with REST APIs and unpredictable data schemas, its architecture complicates write scaling and can impact performance negatively. Conversely, ScyllaDB offers predictable low latency and high throughput, making it more suitable for real-time, latency-sensitive data pipelines and mixed batch workloads, despite its more rigid data modeling restrictions. Alexys Jacob, CTO of Numberly, shares insights from experience, highlighting that while MongoDB is favored for its versatility in storing and querying data, ScyllaDB's consistent performance is preferred for applications requiring strong latency guarantees and schema-based APIs. The coexistence of both databases within Numberly's tech stack exemplifies their complementary strengths, with ScyllaDB increasingly being adopted for specific high-performance use cases, while MongoDB remains valuable for its unique features, provided that its architectural complexities are managed.

Daniel Reis explores the intricacies of data modeling for social media "likes" using ScyllaDB, a NoSQL database, emphasizing the importance of efficient data structures for scalable and low-latency performance. Through his experience preparing a presentation for the CityJS event, Reis discovers that while simple queries like SELECT count(*) can initially work, they become inefficient as data volume grows. To optimize performance, he suggests using ScyllaDB's atomic counter type in a dedicated table to track interactions like likes, thus avoiding the pitfalls of mixing data types and ensuring fast query execution. Reis invites readers to a ScyllaDB Labs event for hands-on experience in building high-performance applications and learning best practices for data modeling and processing.