August 2022 Summaries

The blog post discusses the development and performance improvements of the scylla-rust-driver, an open-source driver for ScyllaDB and Apache Cassandra, written in Rust with an asynchronous API using Tokio. It highlights how the Rust driver outperformed other drivers in benchmarks, leading to its potential use as a unified core for various drivers. The post is based on a ScyllaDB University lesson and covers key topics like Prepared Statements, Paging, and Retries, demonstrating how minor changes can enhance application performance. Prepared Statements involve pre-parsing queries for reuse, reducing database-side calculations. Paging returns large data sets in chunks, improving memory usage and performance, while Retry Policies manage query failures by determining when to retry based on the likelihood of success. The post provides practical examples using ScyllaDB and encourages further learning and certification through ScyllaDB University, emphasizing the benefits of these techniques in building high-performance applications.

Apache Pulsar, an open-source event streaming platform, has gained significant traction since its 2016 release, now standing as a top-level Apache Foundation project with over 500 contributors. The Pulsar Summit 2022 in San Francisco showcased its extensive integration with other technologies and its role in driving data democratization through real-time analytics. Key presentations highlighted Pulsar's integration with systems like Apache Beam and Apache Pinot, underscoring its capability for high-speed data ingestion and processing. The summit also featured discussions on new tools and frameworks such as (pf)SQL for native data transformation within Pulsar, and innovative projects like Kafka-on-Pulsar (KoP), which creates a Kafka-like interface using Pulsar. Speakers from companies like Uber and Mercado Libre demonstrated Pulsar's scalability and low-latency performance in handling massive data streams for real-time applications. The event emphasized Pulsar's growing community and its expanding ecosystem, encouraging further development and collaboration.

ScyllaDB, a NoSQL database solution, is lauded by users for addressing various technical challenges, primarily enhancing performance and scalability in data-intensive applications. Users from diverse industries, including health and wellness, newspaper, and financial services, have transitioned from other databases like MongoDB, Cassandra, and PostgreSQL to ScyllaDB, citing improvements in response times, data ingestion, read performance, and cost efficiency. The database's ability to support a wide range of use cases, from real-time data ingestion and IoT to OLTP and OLAP, has been highlighted as a key advantage, allowing teams to build modular, extendable platforms and future-proof their solutions. Additionally, ScyllaDB's managed infrastructure ensures reliable performance and supports stateful data processing, which significantly reduces costs and complexity. Users have shared these insights on peer review platforms like G2, emphasizing the database's role in overcoming tech debt and supporting innovative projects across various sectors.

ScyllaDB has unveiled its latest version, ScyllaDB Open Source 5.0, marking a new phase focused on innovation beyond compatibility with Apache Cassandra and Amazon DynamoDB. In a TechStrongTV interview, ScyllaDB VP of Product Tzach Livyatan discussed the database's performance and security enhancements, highlighting its ability to maintain high availability even under extreme circumstances, like a data center fire. ScyllaDB distinguishes itself with its second-generation architecture, emphasizing low latency and high throughput, enabling efficient scaling and cost savings. The latest release introduces a new consensus algorithm, Raft, for improved consistency in metadata management, allowing safe and rapid scaling of clusters, even within Kubernetes environments. Additionally, ScyllaDB has optimized its performance for AWS's new I4i instances, achieving over double the throughput and reduced latency compared to previous instances. The company offers ScyllaDB in three forms: open source, an enterprise version with enhanced security features, and ScyllaDB Cloud, a managed database-as-a-service that has gained popularity due to the industry's trend towards cloud solutions.

ScyllaDB, in collaboration with the University of Warsaw, has developed a new project called CQL-over-WebSocket, allowing users to interact with ScyllaDB directly through their web browsers without the need for additional dependencies. Traditionally, interfacing with ScyllaDB required using the Python-based command-line tool cqlsh, which necessitated several dependencies. By leveraging WebSocket, which permits full duplex connections and message streams over TCP, the new system enables a browser-based terminal to interact with ScyllaDB, circumventing the limitations of HTTP and enhancing performance. The project entails a WebSocket server implementation for the Seastar framework, a TypeScript driver for CQL specifically meant for web browsers, and an in-browser cqlsh terminal built with React and Material UI. This initiative, driven by students at the University of Warsaw, not only simplifies user interaction with ScyllaDB but also involves a Bachelor's thesis and aims to contribute back to the open-source community, with ongoing updates and future projects slated for the next academic year.

The Performance Engineering Masterclass for Optimizing Distributed Systems is an educational initiative designed to enhance skills in performance optimization for large-scale, distributed systems. The event, featuring experts from ScyllaDB, Grafana k6, and Dynatrace, offers a comprehensive 3-hour online session, which includes interactive discussions and practical examples focused on SLO-driven strategies to improve system observability and performance from front-end to back-end. Participants are guided through the use of modern tools and methodologies, such as Grafana k6 for load testing and Prometheus for SLO visualization, and learn best practices for managing distributed systems using DevOps approaches. The masterclass culminates in a certification exam, allowing attendees to validate their learning and earn a certificate of completion. The program supports ongoing learning by offering on-demand access to materials for those who wish to revisit the content.

ScyllaDB announced the release of ScyllaDB Enterprise 2022.1, building upon its open-source NoSQL database with enhanced reliability and new enterprise-specific features. This release introduces support for AWS EC2 I4i instances, offering improved performance over previous models, and expands compatibility to Arm-based systems. Key features include Change Data Capture (CDC) for tracking database changes, LDAP integration for authentication and authorization, and granular timeout controls for individual queries. Improvements have been made to the I/O scheduler and SSTable index caching, enhancing performance and reducing latency. The release also includes updates to ScyllaDB's cloud formation and reverse queries, and introduces new virtual tables for configuration management. A new release cycle strategy, incorporating both Long Term Support (LTS) and Short Term Support (STS) releases, aims to deliver features to enterprise customers more frequently. Existing customers can upgrade with support, while a 30-day trial is available for newcomers, alongside educational resources through ScyllaDB University.

The blog post discusses the implementation of a new I/O scheduler algorithm for ScyllaDB to improve the management of mixed read/write workloads, aiming to balance disk throughput and latency. ScyllaDB's I/O scheduler prioritizes requests to prevent latency spikes caused by background activities like compaction and repair, which can interfere with immediate read requests. The new algorithm employs a token bucket model to control the rate of outgoing requests, ensuring they remain within the disk's capabilities while adapting to real-time disk conditions. Testing on ScyllaDB versions 4.6 and 5.0 showed that the new scheduler allows for better handling of high request rates, reducing latency by managing the bandwidth and IOPS more effectively. This advancement will be integrated into ScyllaDB version 5.0, along with additional enhancements such as improved metrics reporting and runtime adjustments to accommodate disk aging and different drive speeds.