January 2021 Summaries

In the January update for Project Circe, ScyllaDB highlights the release of ScyllaDB Open Source 4.3, which marks the general availability of the Change Data Capture feature and introduces an experimental feature for Alternator Streams. This version also involves a significant change by removing the seed node concept in gossip, though it remains in the initialization process. Additionally, ScyllaDB Cloud is now in a beta program for Google Cloud Platform, enabling users to run Apache Cassandra or DynamoDB-compatible workloads. The company also announced the General Availability of the ScyllaDB Operator, allowing Kubernetes management of ScyllaDB clusters, and introduced a new IO scheduler in ScyllaDB 4.4-rc0 aimed at improving performance by balancing bandwidth and IOPS across all shards. These updates were part of the ScyllaDB Summit 2021, with sessions available for on-demand viewing.

During a virtual ScyllaDB Developer Hackathon, a team led by Omri Vardi developed a CQL editor integrated with ScyllaDB Cloud, leveraging Monaco, the code editor behind Visual Studio Code. The team, composed of members from the ScyllaDB Cloud team, aimed to create a tool for writing CQL statements, executing queries, and exploring database schemas in a tree view. They faced decisions between integrating existing open-source projects and developing new solutions, ultimately creating a feature-rich environment with functionalities like autocomplete for CQL syntax, query result management, and schema exploration. Despite challenges, they successfully showcased their project, securing second place in the hackathon, and expressed intentions for future enhancements, such as a possible VSCode extension or desktop application. The hackathon underscored the team's collaborative spirit and anticipation for future projects to benefit ScyllaDB Cloud users.

The text compares ScyllaDB, a NoSQL database, and CockroachDB, a NewSQL database, highlighting their distinct approaches to data management and performance characteristics. ScyllaDB, inspired by Apache Cassandra and Amazon DynamoDB, utilizes an eventual consistency model with a focus on high throughput and low latency, benefiting from its asynchronous, shard-per-core design for resource allocation. In contrast, CockroachDB is based on Google Spanner's concepts, emphasizing strong consistency and SQL compatibility with fully serializable ACID transactions. A benchmark test using YCSB workloads demonstrated ScyllaDB's superior performance, handling significantly more data with higher throughput and lower latency than CockroachDB, which faced challenges with data loading and exhibited higher latency during operations. While ScyllaDB excels in scenarios where maximum speed and scalability are priority, CockroachDB is suited for applications requiring strong transactional consistency and relational database features. Despite their differences, both databases continue to evolve, with ongoing enhancements aimed at improving their respective performance and capabilities.

Coinciding with the virtual ScyllaDB Summit 2021, the ScyllaDB User Awards were announced, celebrating innovative applications of ScyllaDB across various industries. Notable winners included Zillow for their message flow mechanism using ScyllaDB, GE Healthcare for their hybrid cloud approach with the Alternator API, and Disney+ Hotstar for leveraging ScyllaDB Cloud to support their massive streaming service. Mail.Ru was recognized for their cost-efficient data storage solution using HDDs, while Zeotap was awarded for managing a large-scale JanusGraph database with ScyllaDB. Fanatics earned the Humanitarian of the Year award for repurposing their factory to produce PPE during the COVID-19 pandemic. Alexys Jacob from Numberly received the Community Member of the Year award for his collaboration in developing a Python shard-aware driver. The event showcased a range of ScyllaDB use cases, inspiring future innovations in big data solutions.

ScyllaDB Open Source 4.3, released by the ScyllaDB team, introduces several enhancements and new features aimed at improving performance and user experience. This version now includes a production-ready Change Data Capture (CDC) feature for tracking data updates, initially introduced as experimental in version 4.1, and an experimental Alternator Streams API compatible with AWS DynamoDB Streams. The release eliminates the seed node concept in the gossip protocol to simplify operations and introduces a Unified Installer for easier deployment, particularly in environments without root privileges. Additionally, the update provides security enhancements, performance optimizations, such as improved memory usage, and various bug fixes to ensure stability across different functionalities. The release also features new tools and support for Google Cloud Platform (GCP) images, enabling more flexible deployment options.

In response to recent news about terminating its support for Parler, ScyllaDB addresses the situation by emphasizing its commitment to non-partisanship and transparency while explaining the rationale behind their decision. The company, which prides itself on supporting a diverse global user base and maintaining an open-source culture, decided to end its relationship with Parler following the January 6th Capitol siege due to Parler's violation of their license agreement. This decision was influenced by the realization that their technology could potentially facilitate further violence, despite the company's typical focus on technology and business growth rather than customer affairs or free speech debates. ScyllaDB offered migration support to Parler and expressed willingness to work with them again if they aim to foster a safer environment. This experience highlighted the necessity for ScyllaDB to articulate its values clearly and set boundaries against the misuse of its products in ways that could harm democracy, underlining their commitment to preventing violence while remaining open to community feedback.

The ScyllaDB Summit 2021, a successful virtual event, featured a series of presentations from prominent figures such as ScyllaDB CTO Avi Kivity, who discussed the challenges of big data management and the importance of performance optimization in databases. The event highlighted ScyllaDB's capabilities, including Change Data Capture (CDC) and its integration with Apache Kafka, which offers users new paradigms for leveraging big data. Various sessions covered performance enhancements, the use of ScyllaDB in different industries, and practical tips for developing ScyllaDB applications. The summit also included talks on the innovative use of ScyllaDB in video streaming frameworks and distributed ledger technology. The enthusiastic participation of ScyllaDB users and the broader NoSQL community contributed significantly to the event's success, allowing attendees to engage with new technologies and methodologies in data management.

ScyllaDB Summit 2021, held online for the first time, commenced with keynotes from notable figures including ScyllaDB CEO Dor Laor and Jepsen test author Kyle Kingsbury, focusing on ScyllaDB's advancements and future roadmap, particularly Project Circe, aimed at enhancing elasticity, performance, and data consistency. The summit highlighted ScyllaDB's achievements in 2020 such as feature parity with Apache Cassandra, the launch of ScyllaDB Kubernetes Operator 1.0, and the beta deployment of ScyllaDB Cloud to Google Cloud Platform. Presentations covered a wide range of topics including the integration of ScyllaDB's Change Data Capture with Apache Kafka, the implementation of the Raft protocol, and various use cases from companies like Zillow, GE Healthcare, Grab, Disney+ Hotstar, and more, showcasing ScyllaDB's versatility in handling large-scale, distributed database systems. The event featured interactive sessions where attendees could engage with speakers and explore technical insights, culminating in a display of community engagement through a meme contest and discussions around the potential of incorporating CQL into Grafana.

ScyllaDB, a mature NoSQL database, is undergoing a significant transformation through Project Circe, which aims to enhance its consistency, elasticity, and ease of use. This initiative plans to introduce substantial features monthly throughout 2021, including the adoption of the Raft consensus protocol to improve data manipulation and ensure transactional consistency during schema and topology changes. The project addresses the complexities of eventual consistency and aims to make maintenance operations smoother with innovations like Repair Based Node Operations (RBNO). Furthermore, ScyllaDB is focusing on improving elasticity by enabling the addition, removal, and replacement of multiple nodes simultaneously, thereby simplifying cluster management and performance. Additionally, performance enhancements include uninterrupted operation under high loads, improved large partition performance, a new IO scheduler, and the use of coroutines to optimize asynchronous operations. The project also emphasizes stability improvements, such as transactional schema changes and workload shedding algorithms, while expanding deployment options with a Kubernetes Operator and extending ScyllaDB Cloud to platforms beyond AWS.