October 2011 Summaries

Over the past year, significant improvements have been made to Apache Cassandra's performance, particularly in read and write performance. The latest version, 1.0, has seen a 40% increase in write performance compared to the previous release, while read performance has improved by an impressive 400%. These enhancements are achieved through various optimizations such as using lighter-weight data structures for representing row fragments from reads and only deserializing the most recent versions of requested columns. Additionally, improvements have been made to intra-node communication efficiency, resulting in a 15% faster non-local operation speed in Cassandra 1.0.

The Leveled Compaction Strategy in Cassandra is designed to address the shortcomings of the Size-Tiered Compaction Strategy for certain use cases. It ensures that a row is spread across fewer SSTables, reducing read latencies and improving overall performance. However, it requires more I/O for compaction compared to size-tiered compaction. Leveled compaction may be beneficial when rows are frequently updated or if the application has strict latency requirements. It can also save disk I/O in certain scenarios. On the other hand, leveled compaction might not be suitable for write-heavy workloads and could worsen I/O issues if nodes cannot keep up with compaction. Testing with live traffic sampling is recommended before switching to leveled compaction. Additionally, using leveled compaction in combination with SSDs or when the entire dataset fits into memory can provide significant performance benefits.

Cassandra's log-structured storage engine enables its performance and features like application-transparent compression by turning all updates into data files called sstables that are written sequentially to disk. Over time, multiple versions of a row may exist in different sstables with varying sets of columns. To prevent read speed from deteriorating, compaction runs in the background, merging sstables together. Cassandra's size-tiered compaction strategy is similar to Google's Bigtable paper and combines sstables when enough similar-sized ones are present. However, this approach has issues with update-heavy workloads. Cassandra 1.0 introduces the Leveled Compaction Strategy, based on LevelDB from Google's Chromium team. This strategy creates fixed-size sstables grouped into levels, ensuring non-overlapping sstables within each level. Each level is ten times as large as the previous. This approach solves problems with tiered compaction and can be enabled by setting the compaction_strategy option to LeveledCompactionStrategy. While leveled compaction performs roughly twice as much i/o compared to size-tiered compaction, it offers benefits for update-heavy workloads due to fewer obsolete row versions involved.

The text discusses how to deploy and distribute Hadoop components within a Cassandra cluster for efficient processing times. It recommends overlaying Hadoop over Cassandra by installing a Hadoop TaskTracker on each Cassandra node, with one dedicated server for Hadoop components like JobTracker, NameNode, and DataNode. The input/output formats in 0.6 and 0.7 are crucial for Cassandra's Hadoop support, allowing data to be read from and written back into Cassandra. Additionally, the text mentions that work is underway to add support for Pig and Hive, with potential automation of setup tasks in the future.