February 2016 Summaries

The blog post discusses how to choose the most suitable Amazon EC2 instances for running NoSQL databases, with a focus on ScyllaDB. It compares two main storage options available for EC2, Elastic Block Store (EBS) and instance storage, highlighting their advantages and limitations in the context of NoSQL applications. The i2 instance type is favored due to its high performance and efficient use of SSDs, making it a cost-effective choice despite its higher price, particularly for operations requiring high throughput and low latency. In contrast, c3 instances are noted for their better CPU performance but may suffer from higher latency under heavy loads. The post also addresses the "noisy neighbor" problem where shared storage controllers can impact performance unpredictably, suggesting ongoing monitoring and potential instance replacements as solutions. Moreover, the author advises against using instances with GPUs for ScyllaDB and recommends opting for those with "enhanced networking" to ensure consistent network performance.

ScyllaDB, an open-source NoSQL database compatible with Apache Cassandra, announced the release of version 0.18, which focuses on enhancing stability and fixing bugs as part of its incremental improvement strategy in the beta series. Known for its superior performance and consistent low latency, ScyllaDB aims for a general availability release in the first quarter of 2016. Users can access the full list of contributions and known issues on the ScyllaDB wiki and are encouraged to engage with the community via the scylladb-users mailing list for any release-related inquiries. To stay updated on future releases, followers can use ScyllaDB's Twitter or subscribe to their RSS feed.

In the blog post, Tzach Livyatan shares his mixed feelings about using Ansible, highlighting both its strengths and drawbacks based on his experiences managing ScyllaDB and Cassandra clusters on EC2. He appreciates Ansible's straightforwardness, declarative nature, and extensive ecosystem of modules and extensions, which facilitate tasks like stopping services and updating configurations without needing to manage local server lists. However, he expresses frustration with Ansible's additional abstraction layer, which requires learning new commands for tasks that could be handled by existing Unix or Python commands. Despite the learning curve and occasional limitations in flow control, Livyatan continues using Ansible due to its benefits and his investment in understanding its functionality.

CharybdeFS is a new FUSE-based fault-injecting filesystem designed to test how databases, particularly ScyllaDB, handle filesystem errors. It operates as a pass-through layer between a program and a real filesystem, enabling error simulation through a Thrift RPC interface that allows for easy scripting, primarily using Python. By simulating a range of filesystem errors, such as disk IO errors or latency spikes, CharybdeFS aids in ensuring that databases can maintain data integrity and proper functionality under adverse conditions. It has been instrumental in identifying and addressing ScyllaDB's lack of a consistent strategy for handling disk errors, prompting the implementation of a shutdown feature on I/O errors. The tool's flexibility extends beyond ScyllaDB, making it suitable for testing various server software, including other databases and storage systems. CharybdeFS is open-source and available on GitHub, with additional features like distributed and longevity tests for ScyllaDB planned for future updates.

Testing distributed systems like ScyllaDB is a complex task that is being tackled using Jepsen, a tool developed by Kyle Kingsbury for testing such systems. Jepsen is adaptable for testing a variety of distributed databases, including Apache Cassandra, and has been enhanced to test ScyllaDB by incorporating features like faketime, which allows processes in different containers to operate with different time settings. The ScyllaDB team has also introduced a "sidekick" process to simulate higher loads, using cassandra-stress during tests to challenge the system and identify bugs, of which 11 have been filed so far. Future developments include improving Jepsen checkers to verify the state of all nodes at the end of tests and expanding the test suite to cover more complex scenarios, such as schema changes concurrent with network disruptions. The goal is to use Jepsen's flexible framework to rigorously test ScyllaDB under a variety of extreme conditions, thereby enhancing its reliability and performance.

Asynchronous processing is a common practice in networking, but in storage, traditional synchronous I/O methods struggle to keep up with the high IOPS capabilities of modern SSDs. ScyllaDB, a database that combines networking and storage, utilizes the Seastar framework to enable both asynchronous networking and storage operations via a unified API, leveraging libaio for asynchronous I/O commands. However, Linux's limited AIO support requires direct memory access and a suitable filesystem like XFS, which has recently improved its support for asynchronous appending writes—a feature critical for ScyllaDB's immutable SSTables. To ensure compatibility, the fsqual tool was developed to test filesystems for asynchronous appending write support, revealing that while some systems still fall short, others have integrated necessary fixes. This qualification process is essential for ScyllaDB's performance, and future integration with ScyllaDB's distribution will automate this check, thanks to contributions from the XFS team in enhancing Linux's asynchronous I/O capabilities.

ScyllaDB version 0.17 has been released, offering an open-source, Apache-Cassandra-compatible NoSQL database known for superior performance and low latency. This update focuses on enhancing stability and fixing bugs, introducing features such as support for incremental backups, the ability to stop compaction via nodetool, and updates to the latest versions of seastar and scylla-jmx. A comprehensive list of contributions and known issues is available on the ScyllaDB wiki, and the database is on track for a general availability release in the first quarter of 2016. Users are encouraged to engage with the ScyllaDB community through mailing lists and social media to stay informed about future developments.

The blog post discusses the extensive testing procedures employed by ScyllaDB to ensure its stability, correctness, and compatibility with Apache Cassandra. ScyllaDB undergoes several testing layers, including its native tests, Apache Cassandra's unit and distributed tests, and third-party evaluations. The testing process involves functional tests for ScyllaDB packages, longevity testing on AWS, and compliance with the Cassandra Query Language (CQL) standards. The ScyllaDB team has adapted the Cassandra Cluster Manager (ccm) to facilitate ScyllaDB testing and is planning to contribute this enhancement back to the ccm project. The blog also highlights the challenge of ensuring unique timestamps in the development version of ScyllaDB, which is a known bug being addressed. Additionally, it mentions ScyllaDB's continuous integration setup with Jenkins to run tests regularly, inviting enthusiasts of high software quality to consider joining their team. Future posts will explore the use of the Jepsen distributed testing tool to further expand ScyllaDB's testing capabilities.