March 2024 Summaries

ZEE5, the OTT streaming arm of the Indian media company Zee, recently undertook a significant database migration to ScyllaDB to eliminate cloud vendor lock-in, support rapid growth, and achieve a fivefold reduction in costs. The migration was driven by the need for a scalable, multi-region, and high-availability solution that could meet the compliance requirements and the technical pressures of maintaining efficient and cost-effective operations, especially given the challenges of monetizing media consumption in India. ScyllaDB was chosen for its clustered architecture, which facilitated a seamless transition without downtime and allowed Zee to significantly cut the number of database nodes, reducing costs. The migration included moving data for 525 million users and resulted in enhanced performance and lower expenses, earning the approval of Zee's CFO. The successful transition highlighted the benefits of ScyllaDB’s support and local presence, contributing to the efficiency and scalability needed for Zee's future database workloads.

The blog post provides a comprehensive guide on migrating from Amazon DynamoDB to ScyllaDB, detailing each step in the process and offering practical advice for ensuring data consistency and minimizing disruption to live applications. It highlights the use of ScyllaDB Spark Migrator for migrations to ScyllaDB Alternator, a DynamoDB-compatible API, and discusses alternatives for scenarios where the Migrator isn't suitable, such as when using ScyllaDB's Cassandra-compatible API (CQL) or when only streaming a specific set of events. The post covers creating source and destination tables, back-filling historical data via an S3 export, synchronizing the ScyllaDB cluster with DynamoDB using AWS Lambda, and enabling DynamoDB Streams for real-time data updates. It emphasizes the importance of permissions, configuration adjustments for Lambda functions, and the creation of triggers to process DynamoDB changes effectively. Finally, it provides a method to compare records between the two databases to ensure synchronization, while also inviting readers to explore additional resources and case studies of successful migrations.

Natura &Co, a leading global cosmetics brand operating in over 70 countries, relies heavily on ScyllaDB to manage its vast network of over 7 million sales consultants and facilitate its commercial operations. The company embarked on a journey to create a centralized Global Commercial Platform (GCP) tailored to its unique business model, addressing previous scaling challenges and system inefficiencies. Since transitioning from Cassandra to ScyllaDB in 2018, Natura has benefited from ScyllaDB's impressive scalability and low-latency performance, allowing real-time processing of up to 120K operations per second. This capability is crucial for managing extensive data, including 73K indicators and USD 120M in transactions, thereby enhancing decision-making and maintaining strong consultant relationships. ScyllaDB's active-active architecture supports Natura's AWS-based infrastructure, ensuring uninterrupted operations across Latin America and plans for further expansion.

ScyllaDB Cloud, a managed NoSQL Database-as-a-Service built on the ScyllaDB engine, offers enhanced network connectivity solutions compatible with Cassandra Query Language and Amazon DynamoDB APIs, ideal for performance-focused solution architects. It supports both VPC peering and the newly introduced Transit Gateway Connection, which simplifies network management by centralizing connections and routing through a scalable hub-and-spokes model. This feature, added in February 2024, improves connectivity across complex environments by connecting multiple VPCs, load balancers, and VPNs, enhancing security and auditability. Configuring the Transit Gateway in ScyllaDB Cloud involves setting up connections in AWS, with costs incurred for each network component attached per hour and data processing charges based on traffic. Customers can deploy databases using ScyllaDB Cloud in their AWS accounts, optimizing flexibility and cost management, while billing is split between ScyllaDB Cloud and customer AWS accounts depending on the network setup.

ScyllaDB conducted early performance benchmarks on Google Cloud's new Z3 GCE instances, revealing significant throughput improvements compared to the previous N2 generation. The Z3 series, featuring 4th Gen Intel Xeon Scalable processors and substantial local SSD capacity, is optimized for high-density, high-performance workloads requiring low latency. ScyllaDB, a high-performance NoSQL database, demonstrated a 23% increase in write throughput, 24% for mixed workloads, and 14% for reads per virtual CPU compared to N2 instances, despite the Z3's smaller CPU count. These results underscore the potential for substantial performance gains and cost reductions for data-intensive applications, making Z3 a compelling choice for ScyllaDB users.

Patrick's journey into the world of database management is a series of cautionary tales, marked by a series of missteps and lessons learned in his quest to run an online store for green fedoras. After losing his job at a major tech company, Patrick ventured into entrepreneurship, choosing a NoSQL database for his backend based on its trending status rather than suitability for his workload, leading to costly underutilization. His experiences highlight crucial lessons about the importance of selecting the right database for specific workload types, considering the balance between cost and performance in database-as-a-service offerings, and the necessity of observability tools to manage unexpected spikes in traffic. Patrick's story further underscores the critical need for data backups, understanding consistency models, and the implications of Little's Law on system performance. Despite initially overlooking these aspects, Patrick's trials teach the importance of planning for both expected and unexpected spikes, scheduling maintenance during low-activity periods, and leveraging database capabilities to prioritize user requests over background operations.

Migrating from Amazon DynamoDB involves a series of methodical steps to ensure data consistency and continuity, with considerations for aspects like throttling, hard limits, and cost being primary motivators for organizations to seek alternatives. The process typically involves capturing all changes to the source database, copying data to the target database, and replaying any changes to synchronize both databases. Dual-writing is discussed as a potential strategy, but its applicability depends on factors like TTL settings. Back-filling historical data can be approached through methods such as ETL, table scans, or exporting to Amazon S3, each with its own trade-offs in terms of complexity, cost, and time. DynamoDB Streams and tools like AWS Lambda and the DynamoDB Streams Kinesis Adapter allow for capturing ongoing changes to ensure synchronization between databases. The article emphasizes the importance of careful planning and understanding of the migration process, highlighting various AWS tools and methods to facilitate a seamless transition from DynamoDB to other database systems.

ScyllaDB recently hosted a virtual event featuring an "Ask Me Anything" session with its co-founders, Dor Laor and Avi Kivity, where they discussed several advancements and features of ScyllaDB, including the new tablets replication algorithm, async shard-per-core architecture, and caching strategies. They highlighted the efficiency of tablets over Vnodes in scaling operations, as tablets allow for incremental capacity and quicker node integration, reducing the all-or-nothing scaling issue. ScyllaDB's thread-per-core architecture and emphasis on asynchronous operations enable high concurrency and performance, with innovations like automatic tuning and workload prioritization enhancing its adaptability. The database's integrated caching system, which exploits server memory effectively, was contrasted with external caching solutions, offering advantages in performance and cost-effectiveness. With the introduction of S3 support, ScyllaDB aims to optimize storage costs while dealing with the trade-off of higher latency, proposing a tiered storage approach. The discussion also touched on the improvements ScyllaDB offers over Apache Cassandra, such as the absence of JVM, enhanced caching algorithms, and better observability, positioning ScyllaDB as a performant and user-friendly database solution.