August 2016 Summaries

ScyllaDB 1.3 introduces the date-tiered compaction strategy, designed to enhance performance for time-series data by organizing data based on its timestamp to improve read efficiency. This strategy groups SSTables with similar timestamp ranges, reducing the need to consult multiple SSTables during queries, thus optimizing read performance for time-series workloads. Developed initially for Cassandra, it addresses inefficiencies of previous compaction strategies like size-tiered compaction by ensuring that newer data is prioritized and older, less relevant data is compacted efficiently. Despite its advantages, the date-tiered strategy has limitations, such as potential issues with memtable flushing and read repair, leading to the development of the Time Window compaction strategy as a more refined alternative. ScyllaDB plans to support this new strategy to better handle such limitations in the future.

ScyllaDB 1.2.4, a bugfix release of the ScyllaDB 1.2 stable branch, was announced by the ScyllaDB team on August 26, 2016. This release maintains backward compatibility and supports rolling upgrades, addressing several issues such as rare cases of range queries causing core dumps, excessive CPU consumption when compiled with certain versions of Boost, and limitations in performance due to disk bandwidth constraints rather than IOPS. It also resolves problems related to table creation using Thrift with dynamic column families that hindered insertions via CQL. Key contributors to this release include Avi Kivity, Duarte Nunes, Paweł Dziepak, Pekka Enberg, and Piotr Jastrzebski, who made significant improvements such as preserving DMA alignment, ensuring dense schemas are correctly upgraded, fixing iterator invalidation, and addressing access bugs in the storage proxy.

ScyllaDB 1.3 introduced enhanced support for large partitions, addressing the challenges they pose to data modeling and performance. Previously, large partitions could lead to memory issues, cache eviction, and performance degradation, particularly when partitions exceeded the 2 billion clustering rows limit or when dealing with unpredictable data inputs. The update allows the database engine to handle partitions at a clustering row granularity, enabling processes like SSTable compaction and streaming to operate without needing the entire partition in memory. While the new version improves performance by reading only data relevant to query results, there are still limitations, such as handling partitions larger than 128 kB during data streaming and maintaining cache efficiency for large partitions. Tools like nodetool cfstats and cfhistograms aid in monitoring partition sizes, and future updates are expected to further enhance capabilities, addressing current constraints like reversed queries needing entire partitions in memory.

ScyllaDB 1.3 is a production-ready minor release of the open-source NoSQL database compatible with Apache Cassandra, offering high performance and low latency. The release includes several new features such as Thrift support, which allows users of projects like KairosDB and Titan to migrate while maintaining protocol compatibility, and the Date Tiered Compaction Strategy (DTCS) for improved performance in time series use cases. Enhancements also include improved handling of large partitions, CQL tracing for latency analysis, and relaxed polling in Docker support to optimize resource usage. The release introduces internode compression options and support for CQL 3.2.1, alongside a new REST API for dynamically managing metrics. New metrics cover cache, database, IO queue, CQL trace, and Thrift usage, while the update also notes known issues such as CQLSh incompatibility with certain Python versions. The ScyllaDB Summit, coinciding with the Cassandra Summit, will provide further insights into these updates and future developments.

Nick Stott, a software engineer at Compose, discusses his experience with developing application architectures and container orchestration infrastructure, particularly in the context of managing databases like ScyllaDB at an internet scale. During his interview, he highlights the challenges and solutions in adapting the Twelve Factor Manifesto, originally designed for stateless web applications, to work with stateful services such as databases. Stott emphasizes the importance of containers, noting their evolution since 2011 and the need for orchestration layers to manage essential database operations while containers handle configuration internally. He also addresses network security challenges, advocating for private VLANs and portal containers to secure database nodes from public exposure. The interview further explores the complexities of scaling JVM databases and the advantages of ScyllaDB's ability to scale dynamically, highlighting Compose's strategy for handling multi-tenant systems and the role of memory throttling in resource management. Stott expresses enthusiasm for engaging with the community at the ScyllaDB Summit and learning from customer interactions to enhance Compose's offerings.

Henrik Johansson, a Senior Software Developer at Eniro, will be speaking at the ScyllaDB Summit about utilizing ScyllaDB for a microservice-based pipeline in Go. Having transitioned from a physics background to various software development roles, Johansson's work at Eniro involves backend systems for data refinement and analysis using a range of technologies, including ScyllaDB, which was chosen for its speed and SQL-like familiarity. His team opted for ScyllaDB to replace previous systems like Redis and MongoDB, which struggled with performance due to high ID volumes and caching complexities. The switch to ScyllaDB allowed them to remove in-application caches and custom serialization code, simplifying operations and reducing headaches related to cache maintenance and synchronization. Johansson emphasizes the benefits of low-maintenance, high-speed operations with ScyllaDB, which has proved efficient in handling large amounts of data with minimal disk usage. The ScyllaDB Summit will feature various talks, including technical sessions and workshops, and is scheduled to coincide with the Cassandra Summit.

CQL tracing in ScyllaDB is a diagnostic tool that allows developers and administrators to analyze internal data flows within a ScyllaDB cluster, similar to Apache Cassandra's tracing. It helps identify issues such as low replication factors or network delays by providing detailed timestamps and processing information for CQL requests. The tool is particularly useful for diagnosing problems like inefficient node-to-node communication, which can be exacerbated by improper client configurations. Tracing can be resource-intensive, so ScyllaDB offers probabilistic tracing to limit performance impacts by tracing only a fraction of queries. Tracing data is stored in the system_traces keyspace, which includes tables for sessions and events, capturing detailed information about query execution across the cluster. The format of these tables mirrors that of Cassandra, and ScyllaDB continually updates the types of traceable events. The ScyllaDB Summit provides opportunities for further learning and networking among users and developers, offering insights into best practices and future developments.

The ScyllaDB Summit workshop, led by Tzach Livyatan, offers a comprehensive session on setting up, tuning, monitoring, and administering a ScyllaDB cluster, specifically focusing on a 3-node setup on AWS capable of handling high request rates. Participants will gain hands-on experience with tools like CQLsh, nodetool, and scyllatop, learn to identify and understand bottlenecks, and set up a monitoring server. The workshop is part of a larger event at the ScyllaDB Summit in San Jose, which includes discussions on new features, case studies from companies like IBM and Samsung SDS, and networking opportunities with developers and DevOps users. It is scheduled a day before the Cassandra Summit at the Hilton San Jose, and is designed for those with basic system administration and NoSQL knowledge, offering insights into ScyllaDB's design, best practices, and future developments.

ScyllaDB 1.2.3 is a bugfix release from the ScyllaDB team, building on the stable 1.2 branch, and it maintains backward compatibility while supporting rolling upgrades. The update addresses several critical issues, including a bug in the SSTable statistics API that could lead to crashes, a flaw in the Bloom Filter implementation affecting very large SSTables, and a problem where dropping a keyspace with numerous tables might never complete. Additionally, a bug in the io_tune setup step could result in a crash on certain storage configurations. Contributors to this release include Avi Kivity, Duarte Nunes, Gleb Natapov, Pekka Enberg, and Tomasz Grabiec, who have made various fixes and improvements to enhance the reliability and performance of ScyllaDB.