February 2021 Summaries

ScyllaDB University has introduced a variety of new lessons and features as of February 2021, offering comprehensive educational content for developers, DBAs, and architects. Key updates include a Change Data Capture (CDC) feature that facilitates real-time data processing, along with insights into workload prioritization, materialized views, secondary indexes, and lightweight transactions. The platform also covers essential topics such as security, with lessons on authentication, authorization, and encryption, providing a robust framework for managing identities and securing data. Additionally, learners can explore using Spark with ScyllaDB for analytics, delve into the ScyllaDB token ring architecture through new driver lessons, and gain hands-on experience with the Alternator, which is ScyllaDB's DynamoDB-compatible API. Practical guidance is offered for ScyllaDB configuration, repair processes, and monitoring, employing tools like Prometheus and Grafana to ensure optimal cluster performance. These updates underscore ScyllaDB University's commitment to delivering a free, online, and self-paced learning environment that equips users with the skills needed to build and maintain scalable, efficient databases.

Prometheus users often face challenges with recording rules and alerts, as they are generated in real-time and do not apply to historical data, creating difficulties in testing their impact. To address this, a utility called promutil.py has been developed to generate OpenMetrics data, which can be used to backfill Prometheus starting from release 2.25. This allows users to run experiments on separate servers with historical data, helping them assess the effect of new alert rules without affecting production systems. The process involves using promutil.py to produce OpenMetrics files, which are then used to create Prometheus blocks via promtool, enabling the inclusion of historical data in recording rules and alerts. Although promutil.py does not generate historical alerts, it can create corresponding metrics for alert testing and is also useful for ad hoc alert management, offering a valuable tool for system diagnostics and monitoring enhancements.

ScyllaDB's internal hackathon led to the development of a new shard-aware Rust driver for CQL, aiming to fulfill rigorous expectations such as asynchronicity, token-aware routing, and shard-aware routing specifically optimized for ScyllaDB. Existing Rust drivers like cdrs and cassandra-rs did not meet these standards, prompting the creation of an async CQL driver in pure Rust, free from unsafe code. The driver, built using Rust's async/await mechanism and the Tokio framework, supports establishing CQL sessions, executing queries, and fetching cluster topology information, enhancing performance by directing requests to the appropriate nodes and CPU cores. During the development, a bug in the murmur3 hashing algorithm was intentionally replicated to ensure compatibility with Apache Cassandra. Benchmarks indicate superior performance compared to other drivers like GoCQL and cdrs. As the project progresses, a team from the University of Warsaw will continue development, focusing on features like CQL authentication and TLS support, while the official roadmap outlines future enhancements such as CQL tracing and speculative execution.

In February 2021, ScyllaDB University reported a significant increase in user engagement compared to the previous year, with thousands of new users accessing the educational content related to ScyllaDB's features. The blog post highlighted top students from the past year, praising their achievements and sharing testimonials about the effectiveness of the training. Additionally, the ScyllaDB Summit, held virtually for the first time, featured a well-attended Training Day, with all materials published in two courses on ScyllaDB University for developers and administrators. Participants who completed these courses received certificates and ScyllaDB swag, with incentives for sharing feedback and engaging in community discussions via Slack. The post encouraged new learners to create accounts and start free courses, emphasizing the benefits of the training for professional development.

Disney+ Hotstar, originally launched as Hotstar and later rebranded in March 2020, experienced rapid growth in subscriptions, reaching over 300 million users, including both paid and ad-supported tiers. The service set a record for concurrent streaming during the 2020 India Premier League cricket competition. To enhance user experience and manage data effectively, Disney+ Hotstar transitioned from using Redis and Elasticsearch to ScyllaDB Cloud, a decision driven by the need for consistently low latencies and cost-effectiveness. The migration process involved careful data handling, including converting existing data formats and ensuring synchronization with ScyllaDB. This shift allowed Disney+ Hotstar to maintain high performance and scalability without the administrative overhead of their previous systems, enabling them to focus on business growth.

The ScyllaDB Developer Hackathon focused on improving testability by using Docker instead of Kubernetes to streamline the process of running Cassandra Distributed Tests (dtests) with the Cassandra Cluster Manager (CCM). Historically, setting up these tests required complex compilation and configuration steps, which Docker simplifies by providing an isolated environment with network addresses and dependencies pre-installed. The team explored using the Docker-ccm setup for their dtest suite to reduce operational complexity and improve efficiency, with plans to integrate it into their Continuous Integration (CI) processes. While Kubernetes was considered, its automatic cluster management was seen as a barrier to the level of control needed for functional testing. The hackathon also included contributions from various team members across different departments, and ongoing efforts are in place to further optimize the testing infrastructure, including potential use of EC2 spot-fleets for distributed test execution.

Change Data Capture (CDC) in ScyllaDB allows users to track and respond to data changes within a cluster, with ScyllaDB 4.3 marking CDC as production-ready. To facilitate the development of applications that read from ScyllaDB's CDC, libraries for Java and Go have been released, catering to both established Java users and those preferring Go for its concurrency and performance benefits. The CDC mechanism involves CDC log tables, which are linked to base tables and record data modifications such as insertions, updates, or deletions. These libraries abstract the complexities of managing streams and generations, ensuring ordered processing of changes and accommodating topology changes within the cluster. Applications can be written to utilize these libraries, which handle the intricacies of the CDC log format, allowing developers to focus on business logic. The blog post provides guidance on using these libraries, alongside examples and further resources, empowering developers to build efficient, real-time change-tracking applications with ScyllaDB.

ScyllaDB Monitoring Advisor is a new addition to the ScyllaDB Monitoring Stack, designed to emphasize potential problems, configuration issues, and data model suggestions rather than providing a general status overview. Integrated into the overview dashboard, it features an Advisor table that highlights potential issues like large cells and non-prepared statements, and a Balance section that identifies outliers in node and shard behavior. The Advisor employs low-priority alerts and utilizes Grafana's Loki for generating metrics and alerts based on logs, while Prometheus recording rules facilitate metric generation. These enhancements, available in ScyllaDB Monitoring Stack 3.6, aim to improve monitoring efficiency and are accessible for feedback and community interaction through GitHub and Slack.