June 2021 Summaries

Cloning a database cluster using ScyllaDB Manager Backup is a vital process for disaster recovery, system migration, or creating copies for testing and analytical purposes, as described in Michał Matczuk's blog post. The ScyllaDB Manager 2.4 introduces a versatile tool with the download-files command, which simplifies accessing and downloading remote files by replacing vendor-specific tools, enhancing data restoration capabilities. This command allows listing clusters, filtering snapshots, and managing downloads with several customizable options. The cloning process involves creating a new cluster, installing the ScyllaDB Manager Agent, granting access to backups, and using an Ansible playbook to automate restoration. This involves setting parameters such as backup location, snapshot tags, and node mapping in a vars.yaml file. The process includes checking the restoration plan and running the Ansible playbook, with the restoration potentially completing in as little as 15 minutes on a high-speed network. Users can leverage ScyllaDB Manager either as open-source users or enterprise customers and are encouraged to engage with the community for support.

A webinar hosted by ScyllaDB highlighted the integration of ScyllaDB and Confluent Kafka in building event streaming architectures, featuring insights from Numberly's CTO Alexys Jacob and Chief Data Officer Othmane El Metioui, alongside Tim Berglund from Confluent. Numberly employs a hybrid architecture using ScyllaDB and Kafka on-premise and in the cloud, emphasizing the necessity of expertise in managing a multi-datacenter infrastructure. They discussed their transition from using Elastic Beanstalk and RabbitMQ to Redis and Kafka, before ultimately adopting ScyllaDB for low-latency remote state storage to reduce pipeline complexity and enhance performance. They also highlighted the importance of ScyllaDB's Change Data Capture (CDC) feature, which allows for efficient data synchronization and real-time processing. The session underscored how Kafka and ScyllaDB complement each other, especially in handling large data volumes and ensuring low latency, although challenges like managing large partitions and optimizing data models were noted. The integration of ScyllaDB CDC with Kafka Connect was further explored, with a focus on its ease of use and scalability, making it suitable for complex event streaming applications.

At the ScyllaDB Summit 2021, Grab's engineering team members, Chao Wang and Arun Sandu, highlighted how the company leverages ScyllaDB to efficiently manage its large-scale operations across Southeast Asia, which include services like ride-hailing, payments, and food delivery. Chao Wang, focused on Grab's Trust team, detailed the use of a real-time fraud detection system, which employs counters and real-time data processing to maintain user trust and safety. The system's challenges include scalability, manageability, and extensibility, which are addressed through innovations in counter data storage and processing workflows using ScyllaDB and Apache Kafka. Arun Sandu discussed additional applications of ScyllaDB in Grab's operations, such as ad frequency capping and time series data processing, emphasizing the role of ScyllaDB in improving performance metrics and reducing costs. The presentation concluded with examples of optimizations made to manage large data partitions and reduce latency, as well as cost-saving measures through strategic data management practices, which have significantly enhanced Grab's operational efficiency.

The ScyllaDB Operator is a production-ready Kubernetes operator designed for ScyllaDB, a NoSQL database, and was highlighted at the ScyllaDB Summit by engineer Maciej Zimnoch. This open-source project, licensed under Apache, is available on GitHub and supports platforms like Minikube, Amazon EKS, and Google GKE, with the flexibility to be utilized on other Kubernetes providers and on-premises deployments. Initially introduced by Yannis Zarkadas in 2019, the operator has evolved significantly, now offering features like scaling operations, automated configuration rollouts, rolling software upgrades, and auto-healing capabilities. It integrates with ScyllaDB Manager for repairs and backups, supports multi-availability zones, and can manage multiple ScyllaDB clusters within a Kubernetes cluster. Recent updates include the release of ScyllaDB Operator 1.3, enhancing security with dynamic RBAC role assignments. Users are encouraged to engage with the community on Slack and explore further developments or contribute to this rapidly evolving project.

GumGum, a company specializing in contextual advertising, transitioned from using Apache Cassandra to ScyllaDB Cloud due to challenges with Cassandra's performance under high loads and operational complexities. GumGum's Verity platform utilizes natural language processing and computer vision to analyze web content for precise ad placement, reducing reliance on cookie-based methods. Their previous Cassandra setup required significant resources and maintenance, but the switch to ScyllaDB Cloud, a managed system, allowed them to reduce server instances and operational overhead, resulting in cost savings and improved performance. ScyllaDB's architecture, which differs from Cassandra's, required some adjustments in keyspace management and query consistency levels to optimize performance. Despite these initial challenges, the transition led to a more efficient data system, enabling GumGum to focus on developing their ad technology rather than managing extensive database operations.

Project Circe is ScyllaDB's initiative aimed at enhancing its NoSQL database capabilities, with updates shared for May 2021 focusing on operational improvements, performance enhancements, and new features. ScyllaDB has improved failure detection by having nodes directly ping each other and introduced workload shedding based on a new timeout per role feature. Virtual table infrastructure has been enhanced, and off-strategy compaction for repair is now enabled to reduce read amplification. The development of Raft group-0 is progressing, requiring a minimal set of nodes to function, and performance improvements include addressing a range tombstone issue and optimizing memory allocation. Additionally, the ScyllaDB Stress Orchestrator and a cloud-formation-based container client setup are introduced for testing and achieving high request rates. ScyllaDB's Operator 1.2 release for Kubernetes includes autoscaling capabilities, and ScyllaDB Enterprise 2021.1 introduces the Space Amplification Goal for better compaction strategy and new deployment options. Security has been bolstered with the successful completion of a SOC 2 Type II audit for ScyllaDB Cloud, emphasizing the company's commitment to data protection and trust.

ScyllaDB University has introduced interactive labs that provide hands-on experience with ScyllaDB products directly in a browser, eliminating the need for local installation through the use of Katacoda's virtual terminal. These labs, integrated into relevant lessons, cover topics such as quick startup, high availability in a three-node cluster, and basic data modeling concepts like keyspaces and tables. ScyllaDB University is also hosting a free live training event called Summer School on July 28th and 29th, offering sessions tailored to different time zones and expertise levels. This event includes instructor-led training on ScyllaDB basics, advanced topics, and new features, culminating in a roundtable discussion with experts. The training tracks are divided into Essentials, covering installation and basic data modeling, and Advanced topics, focusing on data modeling, application development, and integration with technologies like Kafka and Spark.