April 2022 Summaries

This tutorial demonstrates how to develop an IoT sensor simulator application that streams data to a ScyllaDB cluster on ScyllaDB Cloud, focusing on achieving low latency. It explains the importance of understanding latency types, including end-to-end and application-specific latencies, and emphasizes the need for optimizing internal database processes to handle real-time data from IoT devices efficiently. The tutorial uses a chocolate factory scenario to illustrate how different latency requirements affect system design, highlighting the balance between data processing speed and system constraints. It describes setting up a NodeJS project to simulate sensor data generation, connecting it to a ScyllaDB Cloud cluster, and running batch queries for efficient data insertion. The approach leverages ScyllaDB's sub-millisecond write capabilities and shard-per-core architecture to improve performance by minimizing intracluster communication.

Michał Matczuk's blog post discusses the significant performance improvements achieved by implementing generics in the Go programming language within Google's B-Tree implementation, resulting in a 40% performance gain. B-Trees are self-balancing collections, and while Google's version is already well-optimized, the introduction of generics led to faster execution, reduced memory allocations, and less garbage collection pressure. The partnership with the University of Warsaw facilitated this project, revealing that using generics in Go allows for a zero-allocation API, which minimizes heap size and system resource utilization. The shift from interfaces to generics reduced HeapObjects by 99.53% and improved performance metrics across the board, including a 40% reduction in wall time duration and system resources usage. The blog underscores the potential of Go's generics to enhance performance, especially concerning slices, and highlights ScyllaDB's collaborative efforts with students at the University of Warsaw to push the boundaries of high-performance computing.

The blog post provides a detailed guide on building real-time messaging and streaming applications using ScyllaDB and Apache Pulsar, showcasing the integration of various open-source libraries, schemas, languages, frameworks, and tools. It explains how to deploy an application to a production cloud cluster with StreamNative and create a fast application using Apache Pulsar and ScyllaDB. The post highlights ScyllaDB's compatibility with Cassandra, enabling easy data streaming using Pulsar IO connectors and YAML configuration files. A case study of a chat application is used to demonstrate the process of publishing messages to an event bus, performing sentiment analysis on the payload, and outputting results to a downstream topic. The document also discusses utilizing Pulsar Functions for deploying machine learning models to process messages and includes examples of using JavaScript and WebSockets for data transmission and consumption. The blog concludes by emphasizing the versatility of Apache Pulsar for building streaming applications and encourages further exploration of Pulsar's functionalities through additional resources.

The concept of "phantom jams" in data flow is explored through a comparison to traffic waves, where data slowdowns occur for no apparent reason, despite the absence of bottlenecks. This phenomenon is examined using the Producer-Dispatcher-Consumer model, where a Dispatcher manages message flow between a Producer and a Consumer, potentially causing delays if not optimally configured. Through simulation, it is shown that jitter in the Dispatcher has the most significant impact on system performance, compared to jitter in the Producer or Consumer, affecting both message latency and throughput. The study emphasizes the importance of understanding and monitoring system components to mitigate these effects, especially highlighting the role of the Dispatcher’s effective dispatch rate, which is influenced by dispatch frequency and over-dispatching multiplier. This research underscores the need for careful system architecture and monitoring to prevent inefficiencies in data-intensive applications like those managed by ScyllaDB, a highly performant NoSQL database.

WebAssembly (WASM) is emerging as a suitable backend for user-defined functions (UDFs) in ScyllaDB due to its portability, performance, and secure sandbox environment. ScyllaDB is experimenting with WebAssembly through the Wasmtime engine, which is implemented in Rust and offers a lighter and more asynchronous-friendly alternative to Google's v8 engine. This integration allows UDFs to be written in various languages and compiled into WASM, facilitating complex database operations like User Defined Aggregates (UDAs). ScyllaDB plans to enhance this functionality by developing helper libraries for Rust and C++ and transitioning the UDF layer to Rust to leverage Wasmtime’s asynchronous capabilities fully. This initiative is part of ScyllaDB's broader effort to incorporate WebAssembly into its database infrastructure, aiming to optimize performance and expand functionality.

In April 2022, ScyllaDB announced new free trials for ScyllaDB Cloud, its fully managed database-as-a-service, which is popular among industry leaders like Disney+ Hotstar and Crypto.com for its speed and scalability. The company introduced two distinct trial options to cater to developers and DevOps teams: a 30-day small developer instance and a 48-hour production cluster, both designed to help users evaluate ScyllaDB's capabilities without the usual SLA commitments. Alongside these trials, ScyllaDB launched a new Cloud Documentation Portal and added support for flexible storage on Google Cloud, enhancing customization and cost-effectiveness. ScyllaDB Cloud, known for its close-to-the-hardware, shared-nothing architecture, is now available in 20 global regions and supports real-time applications at a reduced cost compared to other DBaaS options. The platform is API-compatible with Apache Cassandra and DynamoDB, making it a unique offering in the public cloud market.

As storage technology evolves with the advent of fast NVMe devices, the traditional preference for Buffered I/O, where the operating system caches data pages, is being challenged, particularly when contrasted with Direct I/O, which bypasses such caching. Glauber Costa, in his article, argues that Direct I/O offers more reliable and predictable performance by writing data directly to the device, thereby avoiding the pitfalls of delayed costs associated with Buffered I/O, which can lead to increased CPU usage and memory pressure. Using examples from the Glommio io_uring asynchronous executor for Rust, Costa illustrates that Direct I/O not only reduces the illusion of cheaper access created by Buffered I/O but also ensures real-time data persistence without the unpredictability of kernel thread operations. While Buffered I/O might initially appear faster due to its ability to temporarily leverage system memory, it ultimately incurs higher costs and risks as file sizes increase, making Direct I/O the preferred choice for achieving efficient and sustainable storage performance.

ScyllaDB is experiencing significant growth, particularly in its Database-as-a-Service offering, ScyllaDB Cloud, which has seen a 200% increase in growth over recent years and is expected to grow by 140% this year, becoming a substantial part of the company's business. The platform's versatility is highlighted by its use across various industries, including telecommunications, streaming, e-commerce, and crypto, as it addresses the increasing demands of large-scale data management. ScyllaDB's roadmap includes the integration of the Raft consistency protocol to enhance operational efficiency and a move towards a serverless architecture with Kubernetes to improve flexibility, pricing, and elasticity. The company is also seeing a shift from brownfield to greenfield use cases as more businesses choose ScyllaDB for new data-intensive applications. As the database landscape evolves with the growth of digital data, ScyllaDB aims to leverage this trend by offering advanced features such as size-tiered storage and real-time analytical capabilities, ensuring it remains a key player in the database market.