May 2020 Summaries

SAS Institute, a leader in software solutions since 1976, demonstrated a seamless migration of their SAS 360 Match platform from DataStax to ScyllaDB, described as "changing all four tires while driving at full speed," ensuring continuous 24/7 access with no downtime or customer complaints. Their transition, led by David Blythe, involved using an abstract, object-oriented internal API to mask backend database changes, allowing the company to switch vendors without altering code. The migration strategies included lazy writing, lazy reading, and an aggressive lazy reading approach, which minimized data loss and migration time. The switch to ScyllaDB improved read performance significantly, with average read times dropping to low single-digit milliseconds. This migration was executed within a tight two-week window before their DataStax license expired, with no operational disruptions or data loss, showcasing the robust performance of ScyllaDB compared to their previous Cassandra setup.

ScyllaDB Open Source 4.0.1 is a bugfix release for the ScyllaDB 4.0 stable branch, maintaining backward compatibility and supporting rolling upgrades. The release addresses several issues, including an integer overflow in the CQL index reader that could affect queries on large index files, as well as various stability-related concerns such as race conditions during node failures, topology changes, and speculative retries. Additionally, it resolves a stability issue with hinted handoff that could lead to increased memory usage due to invalid hints. The update also fixes a runtime error in the fstrim service within scylla_util.py. Users are encouraged to report any problems encountered with the release.

Investigating issues in distributed systems, such as ScyllaDB, has been simplified by the integration of its internal protocol dissector into Wireshark, a renowned network protocol analyzer. This advancement allows users to scrutinize network traffic between ScyllaDB nodes and clients, converting raw data into a comprehensible format using Wireshark’s filters. This capability supports the analysis of both legacy Thrift protocol and the current CQL, enabling a detailed examination of operations such as reading, writing, schema updates, and gossip information exchange. The dissector, now included in recent Wireshark versions, provides valuable insights into the communication flow of distributed systems, allowing for efficient troubleshooting and performance analysis, especially with the visualization of communication flow graphs. This integration highlights the importance of protocol dissectors in understanding and resolving complex interactions in distributed environments.

Persosa, a Phoenix-based startup founded in 2016, aims to enhance viewer engagement by connecting TV and digital experiences in real time, a capability that requires rapid data processing and minimal latency. After exploring various database options, Persosa chose ScyllaDB over alternatives like Bigtable and MongoDB due to its consistently low latency reads and ease of use, which were crucial for their operations. Initially using ScyllaDB on IBM Compose, Persosa later migrated to ScyllaDB Cloud on AWS, despite their infrastructure being on Google Cloud Platform, as it delivered superior performance and faster response times, significantly boosting customer experience. This transition not only improved request latency—from an average of 80 milliseconds to about 9 milliseconds—but also provided better support and scalability, making it an optimal choice for Persosa's needs in real-time digital advertising experiences.

ScyllaDB Enterprise 2019.1.8 has been released as a production-ready patch focusing on improving the stability and fixing bugs of the high-performance NoSQL database designed for real-time big data workloads. Customers are encouraged to upgrade to this version, coordinating with the ScyllaDB support team to ensure a smooth transition. The release addresses various stability issues, such as memory bloat due to incorrect SSTable metadata eviction, potential race conditions when creating tables with duplicate names, and errors during schema alterations, among others. It also includes performance enhancements like allowing adjustments for slow repairs and ensuring metrics for data streaming are monotonic. This update is part of ongoing maintenance to enhance the reliability and efficiency of ScyllaDB Enterprise for its users.

Zeotap, a customer intelligence platform based in Germany, leverages ScyllaDB and JanusGraph to manage a vast database of 20 billion user IDs, integrating data from over 80 partners and distributing it to more than 40 destinations. Emphasizing consumer privacy and data security, Zeotap's approach involves using deterministic identity resolution to correlate offline CRM data with online identifiers, thereby enhancing the ability to understand customer behavior. They initially used Apache Spark for batch processing but transitioned to a mixed streaming and batch ingestion system with Kafka to meet their service level agreements. This transition was necessary as their database scaled beyond three billion IDs, necessitating more efficient data ingestion and query performance. Zeotap's use of JanusGraph on ScyllaDB allows for low-latency neighborhood traversal and efficient management of linkages, which are crucial for their business model that focuses on transitive links and metadata filtering. The company continuously evaluates and optimizes their system, considering factors like graph database properties, integration with Apache analytics projects, and the operational costs of potential solutions, ultimately choosing JanusGraph backed by ScyllaDB for its cost-efficiency and ability to meet their demanding workloads.

ScyllaDB was developed as an open-source distributed database to improve on Apache Cassandra's performance, retaining compatibility with its APIs and file formats while achieving higher throughput and scalability. It initially adopted Cassandra's CQL API but later introduced Project Alternator, enabling support for the DynamoDB API, which is inspired by Cassandra and Amazon's previous Dynamo work. This blog post compares the CQL and DynamoDB APIs, highlighting differences in their transport protocols, data models, and handling of write operations. CQL uses a binary protocol and requires client libraries to be aware of the cluster's topology, allowing for reduced latency and cost by minimizing network hops. In contrast, DynamoDB uses JSON over HTTP, which is simpler for client libraries but incurs higher latency and costs due to extra network hops and the need for more connections. The post also discusses each API's approach to schema management and write operations, noting that CQL distinguishes between write-only and read-modify-write updates, allowing for optimization of the former. Despite their similarities, ScyllaDB recommends using the CQL interface for most use cases due to its performance advantages, except for migrations from DynamoDB.

ScyllaDB Open Source 4.0 introduces several significant enhancements, including a DynamoDB-compatible API called Project Alternator, improved Lightweight Transactions (LWT), and new experimental features like Change Data Capture (CDC). The release marks a milestone for ScyllaDB, achieving full feature parity with Apache Cassandra, and showcases a focus on performance, usability, and functionality improvements. The development from version 3.0 to 4.0 saw considerable contributions, with 6,242 commits by 80 developers, enabling ScyllaDB to maintain a competitive edge in terms of cost-efficiency and performance compared to Cassandra and DynamoDB. The release also highlights future plans to enhance core performance, improve elasticity, and simplify user experience, positioning ScyllaDB as a cost-effective and scalable alternative for database management.

ScyllaDB Monitoring Stack 3.3 is a newly released open-source monitoring solution for ScyllaDB Enterprise and ScyllaDB Open Source, leveraging Prometheus and Grafana to provide comprehensive monitoring capabilities. This version supports various ScyllaDB releases, including the upcoming ScyllaDB Open Source 4.0 and ScyllaDB Enterprise versions from 2018.x and 2019.x, as well as ScyllaDB Manager versions 1.4.x and 2.0.x, with enhancements like the inclusion of speculative execution details and LWT (Lightweight Transaction) panels to identify performance issues. The update also introduces new dashboards for ScyllaDB OS 4.0 and ScyllaDB Manager 2.1, alongside reorganized CPU and IO dashboards for improved data presentation. Additional features include new Latency alerts and support for Alternator users, as well as various bug fixes and optimizations to enhance system reliability and performance tracking.

Glauber Costa's article discusses two groundbreaking interfaces in the Linux kernel — io_uring and eBPF — that have the potential to revolutionize programming by enhancing performance and flexibility. Introduced in 2019, io_uring is designed to offer fully asynchronous I/O operations, surpassing the limitations of previous interfaces like linux-aio by reducing blocking issues and eliminating the need for thread pools for asynchronous behavior. It achieves this through a shared memory interface that enables efficient batching of system calls and supports a wide range of operations. eBPF, on the other hand, extends the kernel's capacity to execute user-defined code securely and efficiently, enabling advanced performance monitoring and custom functionalities without the overhead of context switches. Together, these innovations allow applications to better utilize modern hardware capabilities, such as multicore processors and fast storage devices, while providing a robust framework for performance analysis and real-time debugging. The article emphasizes their transformative potential for developers, particularly in the context of creating highly efficient, non-blocking applications that can fully exploit the capabilities of contemporary computing environments.