April 2014 Summaries

Software Engineers at Datastax perform a "1000 Node Test" to ensure that their users can trust DataStax Enterprise for handling large-scale "Big Data". This test involves setting up and running operations on a 1000 node cluster, testing standard administrative operations under load. The process includes provisioning nodes, configuring datacenters, inserting data, altering replication strategies, running repair operations, and verifying readability of written data. Challenges encountered during this test include initial provisioning and installation of software, which can be mitigated by using a single base node for installations and leveraging cloud providers' snapshot capabilities to speed up the process. As data continues to grow, Datastax plans to conduct tests on even larger clusters with greater volumes of data in the future.

Cassandra's design to avoid in-place updates and use a commitlog and SSTables for data storage leads to linear disk usage patterns, minimizing seek times. However, this also results in storing the entire history of data changes, which can negatively impact performance when accessing that data. To address this issue, Cassandra's compaction routine reduces the history of changes down to a single set of most recent data per row. Despite this improvement, operating systems use page caches for frequently accessed files, and compaction destroys the original SSTable and creates a new one not in the cache yet. This leads to cache misses during read operations. The patch introduced in CASSANDRA-6916 improves performance by introducing incremental replacement of compacted SSTables, allowing data to be read directly from the new SSTable even before it finishes writing and gradually replacing the old one in the page cache. This results in predictable high performance for Cassandra under heavy load.