In-Memory ScyllaDB, or Racing the Red Queen

Gleb Natapov's blog post explores the challenges of maintaining low latency in Big Data environments, emphasizing the need for constant technological evolution, akin to the Red Queen syndrome from evolutionary biology. As storage technologies advance from HDDs to SSDs and now NVMe drives, the industry must continually adapt to keep pace, with NVMe drives becoming the norm and SATA SSDs viewed as legacy. However, even NVMe can experience high latencies due to factors like SSD garbage collection and concurrent data operations, highlighting the benefits of in-memory solutions for latency-sensitive applications. To address these challenges, ScyllaDB has introduced an in-memory option in its enterprise version, which can significantly reduce latency for certain workloads without needing to rely on disk storage. The post also discusses the trade-offs between using in-memory and regular on-disk tables, depending on an application’s tolerance for latency variability, and provides guidance on configuring ScyllaDB to optimize performance.