December 2023 Summaries

As 2023 comes to an end, a retrospective highlights the top 10 NoSQL blog posts of the year, emphasizing advancements and insights in NoSQL technologies, with a special focus on ScyllaDB. These blogs cover a range of topics, including the introduction of a new book offering guidance on optimizing data-intensive applications, ScyllaDB's shift to a new replication algorithm using tablets for improved elasticity, and comparative analysis of ScyllaDB with MongoDB and PostgreSQL in terms of performance, scalability, and cost. Notable discussions also include Numberly's replacement of Kafka with a Rust-based ScyllaDB application, ScyllaDB's new Raft-based schema management, and intriguing yet overlooked ScyllaDB capabilities. Additionally, the list revisits timeless classics from previous years, showcasing enduring contributions to the NoSQL community, such as performance optimizations in Google's B-Tree implementation and revolutionary programming techniques with io_uring and eBPF.

ScyllaDB Summit 2024 offers a free, virtual, and highly interactive experience for those interested in database performance at scale, featuring keynotes, technical sessions, and hands-on labs over two days. Participants can engage with top engineers from industries like AdTech, social media, entertainment, and technology, learning about various database challenges and solutions such as Disney's migration from DynamoDB to ScyllaDB and Expedia's CDC migration from Cassandra. The summit highlights ScyllaDB's advancements in immediate consistency, extreme elasticity, and upcoming developments related to Kubernetes, compaction strategies, shard-aware drivers, and Change Data Capture. Attendees have opportunities to network with peers and experts, while those interested can binge-watch previous years’ sessions, which include notable discussions on migrating data to ScyllaDB and optimizing database performance.

NoSQL data modeling is crucial for optimizing database performance, particularly in performance-sensitive workloads, and involves avoiding common pitfalls such as misusing collections, creating large partitions, and generating hot spots. Collections should be used cautiously, as they are best for storing small amounts of data, and mismanagement can lead to performance issues. Large partitions, which occur when data grows too big, can cause latency and resource utilization problems, and can be managed by rethinking the primary key and employing techniques like bucketing. Hot spots arise from imbalanced data access patterns, often exacerbated by large partitions, and can be mitigated by adjusting data modeling, refining application access patterns, and implementing rate limits. Understanding and applying these strategies can prevent performance degradation and ensure efficient NoSQL database operations.

ScyllaDB has experienced a transformative 2023, marked by significant architectural advancements, customer success stories, and a focus on elasticity innovation, culminating in a record-breaking P99 CONF and a book release. Originating from a project to build an operating system better suited for virtualization workloads than Linux, ScyllaDB emerged after its founders, Dor Laor and Avi Kivity, recognized inefficiencies in Java-based database implementations like Cassandra. By leveraging modern C++ and a shard-per-core design, they developed ScyllaDB to deliver high performance and scalability. The company has grown to about 160 employees, maintaining a high standard set by its initial team rooted in the open-source community. Looking ahead, ScyllaDB aims to rival DynamoDB in usability while enhancing infrastructure for dynamic workloads, with plans to introduce extreme elasticity in its core product. The ScyllaDB Summit serves as a platform for the community to share insights and learn about ongoing developments, featuring contributions from major industry players like Discord and Expedia.

ScyllaDB Open Source 5.4 has been released, introducing several key features and improvements aimed at enhancing performance and reliability. The update includes Repair Based Node Operations (RBNO), which offers more robust and reliable data streaming for node operations by enabling failed operations to resume from where they stopped. Additionally, the release promotes WebAssembly-based User Defined Functions and Aggregates to a preview status, ensuring backward compatibility while allowing user feedback. Strongly consistent topology updates and schema management, managed by Raft, are now enabled by default, ensuring synchronized metadata across nodes and allowing concurrent topology operations. The update also introduces experimental object storage support, allowing ScyllaDB keyspaces to be stored on Amazon S3 or similar object stores, and adds node-aggregated table-level metrics to reduce the number of metrics. A new guardrail framework is included, providing safeguards against common configuration errors, and a new experimental implementation of nodetool is introduced. Security enhancements include mutual TLS authentication for user access and the ability to preset superuser credentials on boot. Additional improvements span various APIs, performance, stability, monitoring, and deployment processes.

ScyllaDB 5.4 introduces Repair-Based Node Operations (RBNO) as the default method for handling data streaming during node operations, such as adding, removing, and replacing nodes, marking a significant shift from the traditional streaming approach inherited from Apache Cassandra. RBNO utilizes row-level repair to synchronize data between nodes, enhancing performance, consistency, and reliability by ensuring that the latest data is always replicated accurately. This update eliminates the need for post-operation repairs and is complemented by off-strategy compaction and gossip-free node operations, which further optimize the node operation process. Off-strategy compaction speeds up operations by deferring the integration of SSTables until the node operation completes, while gossip-free node operations ensure cluster-wide consistency and allow automatic reversion to previous states in case of errors. These innovations collectively aim to improve data integrity, efficiency, and safety, addressing previous shortcomings in data durability and consistency.

The book "Database Performance at Scale," co-authored by Cynthia Dunlop and colleagues from ScyllaDB, offers a comprehensive guide to optimizing database performance across various platforms, including MongoDB, MySQL, Postgres, Cassandra, and DynamoDB. The 270-page resource, available for free, was collaboratively created without any all-author virtual meetings and aims to address the often-overlooked aspects of database internals, drivers, and infrastructure to help teams achieve low latency and high throughput. Accompanying the book is a masterclass that condenses its key points into a concise format, along with podcast appearances by authors Piotr Sarna and Felipe Cardeneti Mendes, who discuss topics such as database internals, drivers, and the challenges of distributed systems. The book emphasizes practical insights for users experiencing database latency or throughput issues, offering strategies for navigating performance challenges and tradeoffs specific to different workloads and technical requirements.

DynamoDB is a reliable and scalable database solution that's well-integrated within the AWS ecosystem, making it ideal for companies with low throughput requirements or those not concerned with occasional latency spikes. However, as organizations grow, they may encounter challenges such as throttling, high latency during high throughput demands, item size limits, and a lack of flexibility if they wish to adopt a multi-cloud strategy or switch cloud providers. These limitations can lead to significant cost concerns and necessitate complex workarounds, prompting some teams to consider alternatives like ScyllaDB, which offers larger mutation sizes and compatibility with existing DynamoDB data models through its Alternator API. While ScyllaDB isn't a universal replacement for all DynamoDB use cases, it is a strong candidate for teams needing a NoSQL database capable of handling high throughput and providing predictable low latencies. Ultimately, selecting the right database involves careful consideration of current and future needs, ensuring flexibility and scalability as organizational demands evolve.