March 2020 Summaries

Change Data Capture (CDC) in ScyllaDB is an advanced feature designed to facilitate real-time data processing by logging all mutations in selected database tables, enabling immediate reactions to data changes. Unlike Apache Cassandra's approach, ScyllaDB's CDC is more integrated, using a regular table format accessible via the CQL protocol, which simplifies the data capture process and resolves issues related to distributed data systems. CDC logs changes in the form of preimage and postimage data, allowing users to track modifications comprehensively. The feature also supports real-time processing applications, such as Kafka Streams or Spark, and provides seamless integration with other systems for data replication and monitoring. While CDC does impact system performance by increasing write operations, it offers enhanced functionality for processing and analyzing rapidly changing data. The feature, initially introduced in ScyllaDB Open Source 3.2 and improved in subsequent releases, remains in development with plans for general availability in ScyllaDB Open Source 4.1.

ScyllaDB has released version 3.2.4 of its open-source database, focusing on bug fixes and maintaining backward compatibility with the 3.x.y series while supporting rolling upgrades. The release addresses several issues, including incorrect values returned by restricted columns not in select clauses, stability problems causing ScyllaDB to exit under certain conditions like using a replication factor of zero, out-of-memory errors during startup despite available memory, and unwanted population of the system_traces.events even when tracing is disabled. While this update is part of the 3.2 stable branch, users are encouraged to upgrade to the latest stable release, version 3.3, for optimal performance and features.

At the Core C++ 2019 conference, ScyllaDB's CTO, Avi Kivity, presented on the Seastar framework, which optimizes multi-core processing by leveraging asynchronous communications, contrasting it with traditional, synchronous methods that can lead to inefficiencies such as high latency and low CPU utilization. Kivity highlighted the inefficiencies in treating modern hardware like SSDs and multi-core CPUs as synchronous devices, advocating for an asynchronous approach that the Seastar framework employs. This model allows for one thread per core, enhancing performance and avoiding costly locking mechanisms by partitioning tasks across logical cores, or shards, and using a task scheduler. Seastar’s architecture, which integrates seamlessly with C++14 to C++20, enables applications like ScyllaDB to outperform traditional systems like Apache Cassandra by reducing latency and improving throughput. The framework’s design, which includes futures, promises, and continuations, allows for composable APIs that facilitate efficient handling of asynchronous operations, making it well-suited for developing distributed applications. The presentation was part of a broader discussion at the Seastar Summit 2019, where various developers shared insights on leveraging Seastar for building robust, asynchronous applications.

ScyllaDB Open Source 3.3 is a newly released version of the NoSQL database known for its compatibility with Apache Cassandra and superior performance. This release focuses on bug fixes and stability while introducing several experimental features such as Lightweight Transactions (LWT), Change Data Capture (CDC), an Amazon DynamoDB-compatible API called Alternator, and Lua-based User Defined Functions (UDFs). These features are intended to enhance database capabilities, though they remain in experimental stages until further stabilization. ScyllaDB 3.3 also includes enhancements like improved snapshot schema handling, better memory management for large collections, and new client connection information tables, while support for older versions is phased out. The release aims to offer a more resilient and feature-rich database experience, with ongoing improvements and community feedback playing a crucial role in the development of these experimental features before they reach general availability.

ScyllaDB Enterprise 2019.1.6 is a production-ready patch release aimed at enhancing the stability and addressing bugs in ScyllaDB's high-performance NoSQL database tailored for real-time big data workloads. The release introduces improvements such as a new option for the scrub operation to remove corrupt data, enhanced write-path validation tests, and fixes for issues like potential heap-buffer-overflow when stopping the gossip service and a rare race condition in range scans that could lead to data validation errors. ScyllaDB Enterprise customers are advised to coordinate with the ScyllaDB support team for upgrading, and the release is available for both existing customers and as a 30-day evaluation.

ScyllaDB University has made several updates, including the availability of its ScyllaDB Summit training day materials online, offering both novice and advanced tracks to enhance ScyllaDB and NoSQL skills. The platform has introduced new courses, such as ScyllaDB Operations, which focuses on database administration and monitoring, and Using ScyllaDB Drivers, designed for application development using various programming languages. Additionally, an advanced lesson on ScyllaDB Materialized Views and Secondary Indexes has been included in the Data Modeling course. ScyllaDB University also provides recommended learning paths for different roles, such as Administrators, Developers, and Architects, and offers official ScyllaDB certificates upon course completion. All courses are free, online, self-paced, and include practical examples and quizzes, with an invitation for users to join discussions on the #scylla-university Slack channel.

ScyllaDB Monitoring Stack 3.2, released by the ScyllaDB team, is an open-source monitoring solution for ScyllaDB Enterprise and Open Source platforms, leveraging Prometheus and Grafana. This release supports various versions of ScyllaDB and ScyllaDB Manager, offering enhancements like task annotations for ScyllaDB Manager, a new Lightweight Transactions (LWT) section, and detailed I/O queue class displays. The update includes new dashboards for ScyllaDB Open Source 3.3, an upgrade to Grafana 6.6, and Prometheus 2.15.1, along with improved security features like setting a default role in Grafana. Additionally, new features such as BYPASS CACHE panels, local and remote update panels, and a Hinted Handoff panel have been introduced, alongside various bug fixes and annotations for reactor stalls.

The blog post by Michał Matczuk provides a candid exploration of the remote work lifestyle, accentuated by the onset of the COVID-19 pandemic, which has forced many to adapt to working from home. While remote work offers benefits like zero commute time and flexible scheduling, it also presents challenges such as distractions from the home environment, increased risk of burnout, and reduced social interactions with colleagues. Matczuk shares his personal experiences of adjusting to remote work over three years, emphasizing the importance of establishing routines, maintaining physical activity, and managing work-life boundaries to prevent burnout. He highlights the concept of "Perceived Proximity" to foster team cohesion and communication despite physical distance, using tools like Slack and video calls. The post underscores the need for creating a conducive home office setup and finding a balanced schedule while acknowledging the unique pressures faced by working parents, especially during lockdowns with children at home. Overall, remote work demands a proactive approach to managing one’s environment and interactions to maintain productivity and prevent isolation.

Augury, a leader in machine health automation, leverages Industrial Internet of Things (IIoT) technology to provide real-time insights into the condition of industrial machinery, helping prevent downtime and optimize equipment usage. Since its inception in 2011, Augury has expanded globally and formed partnerships with companies like Grundfos to enhance infrastructure safety. The company transitioned from MongoDB to ScyllaDB to better manage its high-volume, multi-dimensional time-series data and facilitate machine learning applications. This switch enabled Augury to streamline data ingestion, processing, and analytics, supporting both real-time and batch processing needs while minimizing operational overhead. Their system includes a machine trend dashboard for immediate data visualization, allowing clients to make informed decisions based on machine health trends. As Augury continues to grow, it is shifting from a microservices architecture to a data pipelining approach with tools like Apache Beam, enhancing its ability to handle increased data flow and complex analytics tasks.

ScyllaDB has released version 3.2.3 of its open-source database, which is a bugfix update for the 3.2 stable branch. This release maintains backward compatibility and supports rolling upgrades, ensuring smooth transitions from previous versions. The update addresses several issues, such as a Debian/Ubuntu upgrade failure, a potential heap-buffer-overflow when stopping the gossip service, incorrect IO class utilization in commitlog operations that may cause delays, and errors during node shutdown. Users are encouraged to report any problems they encounter to assist in further improving the platform.

ScyllaDB Open Source 3.1.4 is a bugfix release for the ScyllaDB 3.1 stable branch, ensuring backward compatibility and supporting rolling upgrades. This release addresses several issues, including correctness problems with hinted handoff sending counter hints as updates when targeted nodes are absent, and incorrect encoding of negative varint values. It also resolves security concerns with reading YAML from Python and various stability issues like commitlog IO class errors, invalid time UUID causing exits, and crashes during initialization. Performance improvements include preventing cache invalidation when no sstable is added during streaming. A specific upgrade note advises users who installed ScyllaDB 3.1.0 fresh to enable compatibility mode in the configuration file before upgrading to 3.1.4. The release is part of the ongoing support for the 3.x.y series, with plans to discontinue 3.1.x support after the release of version 3.3.

Tubi, an ad-supported video on demand (AVOD) platform, has successfully leveraged a NoSQL database, ScyllaDB, and the Scala programming language to enhance its machine learning and personalization infrastructure. Initially using a monolithic architecture with Apache Spark and Redis, Tubi faced challenges in scalability and efficiency, particularly in running A/B experiments, which required extensive manual processes and collaboration between machine learning engineers and backend teams. The transition to a new architecture, featuring a Ranking Service and an experiment engine named Popper, streamlined these processes by embedding ScyllaDB and using AWS Kinesis and Apache Airflow for automatic model deployment. This new system, based on Scala/Akka applications, reduced latency, improved maintainability, and allowed engineers to conduct experiments without code changes or manual intervention, thereby optimizing machine learning operations and enhancing user experience.