September 2022 Summaries

A recent online masterclass on optimizing distributed system performance brought together experts Henrik Rexed, Leandro Melendez, and Felipe Cardeneti Mendes, who shared their insights on performance engineering and load testing. The sessions, now available online for free, covered strategies from front-end to back-end optimization using Service Level Objectives (SLOs) and Service Level Indicators (SLIs), with practical examples featuring tools like Grafana k6 and Prometheus. Participants learned about automating deployments with Keptn and diagnosing distributed database performance issues through observability techniques. The experts emphasized the importance of understanding metrics within the context of specific applications, acknowledging that no universal standard exists for performance measurement. They highlighted the necessity of historical data analysis to identify patterns and deviations in system performance. The sessions concluded with a panel discussion addressing key performance engineering questions and trends, inviting viewers to explore these topics further through the on-demand masterclass and certification exam.

Avi Kivity, ScyllaDB's CTO and co-founder, has pioneered a method to gain deep insights into the operations of NVMe SSDs, particularly in the context of ScyllaDB's unique close-to-the-hardware architecture, which he presented at P99 CONF. By visualizing real-time IO operations, including reads, writes, and disk bandwidth usage on AWS EC2 I4i instances, Kivity elucidates how ScyllaDB's shard-per-core architecture and other database operations are reflected in disk behavior. He further explores how these operations manifest on monitoring dashboards, providing an in-depth look at database internals. The P99 CONF also features additional talks on various low-latency engineering topics, such as analyzing virtual machine overhead, the use of user-mode threads for concurrency, optimizing performance with variable payloads, and maintaining low latency in user-defined functions using WebAssembly.

The transition from batch-oriented processing to event streaming architectures is a notable shift in the current tech cycle, as databases need to accommodate real-time data changes at high throughput rates. This shift emphasizes the importance of selecting a database that aligns with event streaming, considering factors such as cloud-native capabilities, intrinsic qualities like reliability and performance, event-driven design, and the best fit for specific use cases. ScyllaDB, a highly scalable NoSQL database, aligns well with event streaming systems such as Kafka due to its low latency, high throughput, and compatibility with Cassandra CQL, although it lacks native connectors for Apache Pulsar. This adaptability allows ScyllaDB to integrate effectively with event streaming platforms, offering advanced Change Data Capture (CDC) capabilities, although there are opportunities for further optimization in connector development for easier deployment and enhanced performance.

In a recent podcast, ScyllaDB CEO Dor Laor and SADA CTO Miles Ward discuss their enthusiasm for leveraging modern cloud infrastructure, highlighting ScyllaDB's evolution from its roots in KVM to a powerful alternative to Cassandra and DynamoDB. The conversation explores ScyllaDB's establishment as a faster, more efficient database, emphasizing its open-source origins and its transition to a Database as a Service (DBaaS) model, which aligns with market trends. ScyllaDB's reliability and speed are trusted by major platforms like Discord, underscoring its significance in supporting business-critical operations. The discussion also touches on the technical challenges and responsibilities involved in ensuring ScyllaDB's performance, with insights into the company's commitment to addressing customer needs and resolving technical issues. Additionally, the podcast delves into ScyllaDB's collaboration with Google Cloud and SADA, highlighting features like consistency, elasticity, and a DynamoDB-compatible API that facilitates integration with GCP and on-prem environments.

Zeotap, a customer intelligence platform focused on European marketers, transitioned its workload to ScyllaDB after exploring other technologies, driven by the need for GDPR compliance and the impending phase-out of third-party cookies by Google. Originally utilizing Google BigQuery, Zeotap faced challenges with sub-second latency requirements for real-time data processing. Subsequent efforts with JanusGraph and HBase addressed some issues but lacked full control over data models and adequate enterprise support. Ultimately, Zeotap adopted ScyllaDB, which met their technical demands, including sub-second latencies, enterprise support, and a mature monitoring stack. This allowed Zeotap to enhance its CDP capabilities, providing a more efficient and privacy-compliant data management solution while supporting complex customer journeys and personalized marketing experiences.

The blog post discusses recent benchmarks comparing ScyllaDB 5.0 on AWS i4i.4xlarge instances to ScyllaDB 4.4.3 on i3.4xlarge instances, revealing significant improvements in performance for the newer setup. These benchmarks highlight that the i4i instances, equipped with Intel Xeon Scalable Processors and AWS's Nitro SSDs, deliver superior throughput and reduced latency compared to the i3 instances. Specifically, the i4i instances achieved up to 2x higher throughput for both disk and memory-intensive workloads across various scenarios, including write-only, read-only, and mixed read/write tasks. The study also delves into administrative operations like "Add Node" and "Major Compaction," noting unexpected latency improvements during compaction. The findings suggest notable efficiency gains with ScyllaDB 5.0 on i4i instances, encouraging further benchmarking and comparisons against other database systems for broader insights.

Happn, a popular dating app launched in 2014, has experienced significant growth, reaching over 128 million users by 2022, due to its unique approach of connecting singles who have physically crossed paths, emphasizing authenticity in online connections. Initially using Apache Cassandra to handle its data needs, Happn faced challenges related to technical debt, cost, and data management, prompting a shift to ScyllaDB after an internal analysis. This migration, which involved moving fourteen clusters and addressing complex counter data issues, resulted in improved performance and cost efficiency, reducing latencies and storage requirements significantly. The transition to ScyllaDB allowed Happn to scale its operations, optimizing its infrastructure from 48 Cassandra nodes to just 6 ScyllaDB nodes, and cutting its total cost of ownership by 75%. Happn's successful adoption of ScyllaDB, alongside companies like Tantan, Discord, and Rakuten, highlights the database's capacity to support large-scale, high-performance applications in various sectors.