June 2022 Summaries

ScyllaDB has introduced a novel repair-based tombstone garbage collection (GC) method as an experimental feature in its Open Source 5.0 version to address the issue of data resurrection. This approach departs from the traditional timeout-based tombstone GC, which relies on the gc_grace_seconds parameter to determine when tombstones should be purged. The repair-based method ensures that tombstones are removed only after a repair is conducted, eliminating the pressure to complete repairs within a specific timeframe and enhancing data consistency by preventing the reappearance of deleted data. This new method also reduces the need for administrators to determine an appropriate gc_grace_seconds value and allows for more flexible repair scheduling, potentially improving read performance and reducing latency impacts on user workloads. Users can activate this feature through new table configuration options, ensuring greater control over tombstone management and overall database safety.

Disney+ Hotstar, a leading streaming service with a substantial portion of Disney+'s global subscribers, faced challenges with its "Continue Watching" feature due to a complex data infrastructure initially built on Redis and Elasticsearch, which struggled with scaling and cost efficiency. To address these issues, the team, led by Vamsi Subash Achanta and Balakrishnan Kaliyamoorthy, redesigned their data model using ScyllaDB Cloud, a high-performance, low-latency database-as-a-service. This transition simplified their architecture, reduced latency for data reads and writes, and decreased administrative burdens, allowing them to manage the rapidly expanding user base and content library more effectively. The migration process involved converting data from Redis and Elasticsearch to ScyllaDB Cloud without downtime, ensuring a seamless user experience even during periods of high traffic. This strategic move has enabled Disney+ Hotstar to focus on enhancing user engagement and developing new features, such as platform recommendations and watchlist functionality, further solidifying its position as a major player in the streaming industry.

Alexys Jacob, the CTO of Numberly, a French digital data marketing company, embarked on a significant technical challenge by transitioning key components of their data processing pipeline from Python to Rust to enhance performance with ScyllaDB and Apache Kafka. Despite initial resistance due to the lack of Rust expertise at Numberly, Alexys was driven by Rust's promise of creating reliable, efficient software and its advantageous features such as strong type safety, comprehensive error management, and improved dependency management. Although Rust was not faster to develop or prototype than Python, it was chosen for its potential to provide faster data processing, driven by the need to innovate and ensure reliability in their event streaming applications. Alexys highlighted that adopting Rust required overcoming a learning curve, but its long-term benefits in software reliability and performance justified the transition. The project involved integrating the Rust data processing application into their existing infrastructure, including Kubernetes, Prometheus, and Grafana, marking a high-stakes move straight into production rather than a simple exploratory project.

ScyllaDB University LIVE is an upcoming virtual training event scheduled for July 28th, 2022, offering free, instructor-led sessions focused on Scylla and NoSQL databases. The event features two parallel tracks catering to both beginners and advanced users, covering topics such as Scylla architecture, data modeling, Kafka, Change Data Capture (CDC), and deploying ScyllaDB on Kubernetes. Attendees can switch between tracks and engage with leading experts through interactive sessions, with opportunities to ask questions and participate in an expert panel. Following the sessions, participants can further their learning through ScyllaDB University courses, which provide hands-on experience and certification opportunities. The event aims to offer a comprehensive understanding of both foundational and advanced ScyllaDB technologies, with additional incentives like exclusive swag for participants.

Palo Alto Networks, a global leader in cybersecurity, processes terabytes of network security events daily and sought a solution to correlate these events in near real-time without the operational overhead of deploying an additional message queue system like Kafka. The engineering team opted to replace Kafka with ScyllaDB, their existing low-latency distributed NoSQL database, to serve both as an event data store and a message queue. This approach allowed them to streamline operations by eliminating Kafka, reducing costs, and maintaining high throughput performance. The system designed by Principal Software Engineer Daniel Belenky and his team involves the ingestion of disparate events from various sensors, normalization of data into a canonical form, and the use of ScyllaDB to shard the data for parallel processing by multiple worker components. This architecture succeeded in achieving the project goals while minimizing complexity and operational costs, highlighting the potential of ScyllaDB for other organizations facing similar challenges in stream processing and event correlation.

The Distributed Data Systems Masterclass, hosted by ScyllaDB and StreamNative, is a free interactive online course designed to address the challenges of modern data synchronization and real-time event streaming, shifting from traditional batch processing to real-time data propagation with minimal delays. Scheduled for June 21, 2022, the course will feature industry experts Maheedhar Gunturu, Tim Spann, and Raouf Chebri, who will cover topics such as microservices, serverless architectures, and event streaming technologies like Apache Pulsar. The masterclass aims to provide participants with a comprehensive understanding of distributed data systems, offering practical insights into integrating low-latency databases with high-throughput event streaming platforms. Attendees will also have the opportunity to engage in discussions, ask questions, and earn a certification by passing an examination at the event's conclusion.

ScyllaDB is introducing a new Enterprise Feature Release alongside its existing Long Term Support (LTS) release strategy to align more closely with the Open Source version's latest features. This new approach allows customers to choose between a Short Term Support lane, which provides rapid access to new features, and a Long Term Support lane, which offers stability and annual major updates with only urgent bug fixes. The Enterprise Feature Release will occur 3-4 times a year, incorporating both Open Source and exclusive Enterprise features. Semantic versioning will continue to be used, with a major, minor, and patch release format, where patches focus solely on bug fixes. The change aims to prevent the Enterprise version from lagging behind the Open Source version, offering both flexibility and up-to-date functionality for users. Existing support for the two most recent major LTS versions will remain unchanged, ensuring continued stability for long-term customers.

Rakuten, a leading Japanese online shopping platform, faced challenges with volatile latencies while using Apache Cassandra for their catalog system, which handles massive data from numerous vendors and requires real-time updates for optimal shopping experiences. To address these issues, Rakuten migrated to ScyllaDB, which offered the same horizontal scalability but with more consistent performance due to its C++ base and Seastar framework, reducing kernel-level contention and improving parallelism. The transition led to a significant improvement in their data processing capabilities, including a 30% to 35% increase in product feed ingestion rates and a 2.5 to 5 times enhancement in publishing enriched data to partners. This migration also allowed Rakuten to reduce their cluster size from 21 Cassandra nodes to just 6 ScyllaDB nodes, resulting in lower total cost of ownership due to hardware savings and easier cluster administration. ScyllaDB's Incremental Compaction Strategy and shard-aware drivers further contributed to these performance gains, providing Rakuten with a more efficient and reliable system to support their vast and rapidly growing catalog.