Incremental Compaction 2.0: A Revolutionary Space and Write Optimized Compaction Strategy

Incremental Compaction Strategy (ICS) has evolved with the introduction of the Space Amplification Goal (SAG) in ScyllaDB Enterprise 2020.1.6, designed to optimize both space and write performance for overwrite-intensive workloads. Initially, compaction strategies like Size Tiered (STCS) and Leveled (LCS) had limitations, with STCS favoring write performance but suffering from space and read amplification, while LCS excelled in space efficiency at the cost of write amplification. ICS addressed some of these issues but still faced challenges with space amplification under overwrite conditions. The new SAG feature allows ICS to maximize disk utilization without compromising write performance by implementing a hybrid approach that merges the best aspects of both leveled and size-tiered structures, particularly optimizing the largest tier for space efficiency. This makes ICS + SAG, or ICS 2.0, an attractive option for users dealing with overwrite workloads who prioritize space utilization. The configurable SAG parameter enables users to balance space and write amplification according to their specific needs, offering a more efficient alternative to LCS for certain workloads.