January 2024 Summaries

At the ScyllaDB Summit, discussions on WebAssembly (Wasm) highlighted its transformative potential in the data and database sectors, particularly through talks by experts Brian Sletten and Piotr Sarna. Sletten emphasized Wasm's ability to challenge traditional assumptions about data storage and code execution, advocating for a model where applications and databases can operate more cohesively across various environments, from cloud to edge computing. Sarna introduced libSQL, a fork of SQLite designed for distributed systems and edge computing, enhancing offline-first applications through Wasm-powered dynamic functions. The summit underscored a shift from centralized "web scale" big data to decentralized small data systems, allowing more flexibility and efficiency in data handling and application deployment. The conversation also touched on the evolving landscape of programming languages, with Rust and LLVM emerging as key players in optimizing performance and safety. These discussions point to a future where data and computational architectures are more fluid and adaptive, driven by business and technical needs.

India's largest homegrown social media platform, ShareChat, with approximately 180 million monthly users, has successfully leveraged Kafka streams and ScyllaDB to handle vast quantities of user engagement data, achieving sub-millisecond latencies even at over a million operations per second. The platform, which supports content creation in 15 languages, aggregates user interactions such as likes and shares to enhance user experience and content curation. To manage the high volume of engagement events efficiently, ShareChat selected stream processing over request-response and batch processing due to its ability to offer continuous, non-blocking data processing. The architecture employs Kafka for event capture and processing, with ScyllaDB providing the necessary low-latency database support. The transition to ScyllaDB has not only improved latency but also reduced database costs by over 50%, offering detailed monitoring capabilities that enhance operational transparency. This strategic shift has significantly optimized ShareChat's application performance, enabling them to meet the demands of rapid business growth and providing a more responsive user experience.

ScyllaDB is a high-performance, distributed NoSQL database that thrives on its open-source, community-driven nature, which plays a crucial role in its development and enhancement. The ScyllaDB community is vibrant, facilitated through platforms like the Community Forum and Slack, where developers, engineers, and users globally contribute to its growth through discussions, bug reports, and feature requests. The Community Forum, hosted on Discourse, allows for in-depth technical discussions and serves as a repository for community-generated content, while Slack provides real-time communication and informal conversations. ScyllaDB also organizes events such as the ScyllaDB Summit and P99 CONF, offering opportunities for learning, collaboration, and recognition of contributors, thereby fostering a sense of community and encouraging active participation.

DBeaver, a universal database manager that supports both SQL and NoSQL databases, now offers seamless integration with ScyllaDB, a high-performance NoSQL database, as part of its Enterprise version 23.3 release. This update provides users with a graphical user interface (GUI) alternative to the command-line tool cqlsh, featuring syntax highlighting and the ability to connect to both self-hosted and ScyllaDB Cloud instances. Leveraging DBeaver's compatibility with Apache Cassandra, users can easily set up connections to ScyllaDB and perform various queries, including creating new database objects and running complex CQL queries. The tool is available on Windows, Linux, and MacOS, facilitating database management across multiple platforms and enabling users to visualize data and manage schemas effectively.

ZeroFlucs, a company specializing in processing sports betting data, employs ScyllaDB to optimize data replication for low latency and high throughput, crucial for their complex same-game pricing models. At ScyllaDB Summit 23, ZeroFlucs’ Director of Software Engineering, Carly Christensen, explained the company's architecture, which utilizes Kubernetes with over 130 microservices, including custom resource definitions for easy scalability. ZeroFlucs processes approximately 250,000 in-game events per second, necessitating a database that supports high concurrency and low latency across global regions. After evaluating several databases, they chose ScyllaDB for its distributed architecture, allowing data to be located near services and customers to ensure minimal latency. They utilize ScyllaDB's flexibility in resource allocation and replication strategies, dividing data into global, regional, and customer-specific categories to ensure data locality and resiliency. To manage the orchestration and updating of keyspaces, ZeroFlucs developed Charybdis, an open-source Golang library that simplifies CRUD operations and replication management. Carly notes their ongoing journey with ScyllaDB, highlighting challenges such as dynamic keyspace creation and the potential for further performance improvements.

ScyllaDB Summit 2024 showcased a wide array of database insights and technical expertise from industry leaders like Disney, Discord, and Expedia, focusing on addressing database challenges such as latency, throughput, and cost. The virtual event featured keynote speeches and discussions on topics like AI/ML, Kafka, and database internals, with a specific emphasis on ScyllaDB. Attendees had the opportunity to participate in hands-on training sessions tailored for both newcomers and advanced users, covering topics such as data modeling and high-performance application building. The summit also highlighted ScyllaDB's engineering advancements, including new replication algorithms and performance optimizations, and provided access to books authored by industry experts, offering comprehensive knowledge on ScyllaDB and database performance.

Coralogix, an observability platform, dramatically improved its query processing speed by migrating from PostgreSQL to ScyllaDB, reducing processing times from 30 seconds to 86 milliseconds. The move was motivated by the need to handle large-scale, semi-structured data efficiently, as the original PostgreSQL-based metastore implementation could not meet the low-latency and scalability requirements. By switching to ScyllaDB, Coralogix was able to improve performance significantly, handling 10,000 writes per second with sub-millisecond latency and processing 50,000 Parquet files in less than 500 milliseconds. The migration involved reworking data modeling to accommodate ScyllaDB's NoSQL architecture, focusing on partitioning and read/write patterns. Despite challenges such as high tail latency in column metadata processing and managing large partition sizes, the transition was completed in just two months, aided by tools like the ScyllaDB Operator for Kubernetes. Future plans include leveraging WebAssembly User Defined Functions to further optimize data processing.

In a blog post by Attila Tóth, the creation of a video streaming app is detailed, highlighting its use of ScyllaDB and NextJS. The app, designed for minimalism and essential functionality, allows users to list, watch, and continue videos with progress tracking. The technology stack includes TypeScript, ScyllaDB, NextJS, and Material-UI. ScyllaDB is emphasized for its low-latency performance, crucial for video streaming, and the data modeling process is explained, focusing on queries and schema design. The post further delves into specific page queries, the use of user-defined functions, and provides instructions for setting up the app, while encouraging users to explore ScyllaDB University for more educational resources.

ScyllaDB employs a specialized row-based cache that completely bypasses the Linux page cache to enhance performance, particularly for real-time and latency-sensitive applications. Built on the Seastar framework, ScyllaDB's architecture leverages a shard-per-core and shared-nothing asynchronous model, enabling low-latency reads and providing users with comprehensive visibility into cache operations such as hits, misses, and evictions. By not relying on the Linux page cache, which is less efficient for database-specific needs and can lead to high read amplification due to its 4KB chunk size, ScyllaDB's internal cache allows for dynamic tuning according to workload demands, avoiding the complexities associated with multiple caching mechanisms found in Apache Cassandra. The system's object cache design optimizes memory usage and CPU overhead by storing actual row objects rather than 4KB buffers, and it is integrated into ScyllaDB's thread-per-core architecture, which ensures efficient data handling without the need for complex concurrency controls. This approach not only improves performance but also reduces the need for external caches, making ScyllaDB a more streamlined and cost-effective solution.

To reduce the total cost of operating a highly-available database, teams can adopt three technical shifts: moving to fewer, larger nodes, allowing the database to manage itself, and consolidating workloads under a single cluster. This approach addresses various cost factors beyond just database and infrastructure costs, including operational expenses related to tuning, monitoring, and administration. The first shift involves reducing node sprawl by migrating to larger nodes, which reduces server costs and administrative overhead, as demonstrated by a ScyllaDB customer who achieved a 10x reduction in costs. The second shift focuses on minimizing manual tuning by enabling the database to prioritize activities automatically, which reduces the risk of failures and the need for constant supervision. The third shift suggests running multiple workloads on a single cluster while managing resource contention through isolation strategies, such as physical, logical, and scheduled isolation. These shifts help organizations optimize performance and reduce costs, as shown in examples of ScyllaDB's workload prioritization capabilities, which balance OLAP and OLTP tasks without compromising the overall system performance.