August 2020 Summaries

In a blog post by Piotr Sarna, the development of an asynchronous userspace file system, SeastarFS, is detailed as part of a student project sponsored by ScyllaDB in collaboration with the University of Warsaw. The project aimed to create a log-structured file system to meet ScyllaDB's specific latency requirements and to reduce dependency on third-party software like XFS. Implemented in Seastar, this file system allows for better control over memory usage and latency guarantees, aligning with ScyllaDB's shared-nothing design. SeastarFS is designed for high throughput and low latency, optimized for large files and sequential read/write operations, diverging from traditional POSIX compliance by focusing on performance over comprehensive features. The blog also discusses the challenges of compaction in log-structured file systems and the intricacies of lock management to maintain metadata consistency. Although not yet production-ready, the project has shown promising test results when compared to existing systems like XFS, and its development has contributed to the academic work of participating students.

ScyllaDB Enterprise 2020.1.0 is a major release that builds on the features of ScyllaDB Open Source 4.0, introducing enhancements such as a DynamoDB-compatible API called Alternator, which allows seamless migration with no application code changes. This release offers improvements over Apache Cassandra with more efficient Paxos consensus for Lightweight Transactions (LWT) and advanced features in the Cassandra Query Language (CQL), including BYPASS CACHE and LIKE operators. Unique innovations like the Incremental Compaction Strategy (ICS) and Workload Prioritization enable better resource utilization and operational efficiency. ScyllaDB Enterprise 2020.1.0 supports expanded deployment options, including ScyllaDB Cloud, Docker, and Kubernetes, providing flexibility and cost savings over traditional platforms. The update aligns with industry advances in cloud computing and operating systems, and it includes comprehensive enhancements in performance, security, and usability, such as encryption at rest and improvements in query handling. The release encourages users to upgrade from previous versions to leverage new features and optimizations for real-time big data workloads.

The blog post discusses the integration of IOTA's distributed ledger technology, the Tangle, with ScyllaDB through the permanode software, Chronicle, to enhance real-world applications. IOTA's Tangle, unlike traditional blockchains, uses a directed acyclic graph structure, allowing for transaction validation without fees or miners, and aims for complete decentralization through the Coordicide project. The IOTA Foundation, a non-profit, seeks to advance IOTA technology by fostering community engagement and developing open-source software. Permanodes like Chronicle are vital for storing extensive data histories, crucial for use cases that require long-term data retention without burdening network nodes. ScyllaDB is chosen for its fault tolerance, data consistency, and efficient query capabilities, supporting Chronicle's function of storing transactions from the IOTA network. Chronicle's architecture, with components like the dashboard, broker, and storage, facilitates seamless data handling and integration with IOTA projects. Future enhancements for Chronicle include a UI dashboard and selective transaction storage, with documentation and source code available for community collaboration.

In 2019, ScyllaDB supported a student program at the University of Warsaw, where undergraduate Computer Science students collaborated with ScyllaDB engineers to enhance ScyllaDB and its Seastar engine, with projects such as implementing Parquet support. The project aimed to adapt the Apache Parquet format for use with ScyllaDB, which traditionally uses SSTables, to offer more efficient data storage options. The students developed a new library, parquet4seastar, from scratch to integrate Parquet with Seastar's asynchronous framework, facilitating low-latency operations. They also created a demo application, parquet2cql, which translates Parquet files into CQL queries for ScyllaDB. Testing showed promising results in terms of performance and storage efficiency, particularly where Parquet outperformed SSTables in scenarios with fewer unique data values. The work not only contributed to the students' Bachelor’s thesis but also laid the groundwork for future collaborations between ScyllaDB and the University of Warsaw.