Worry-Free Ingestion: Flow Control of Writes in ScyllaDB

In a talk at the ScyllaDB Summit 2018, Nadav Har'El discussed strategies for managing data ingestion in ScyllaDB, focusing on maintaining optimal write speeds without overwhelming the database. ScyllaDB uses flow control mechanisms to balance the rate of data writes, preventing server overload by using both throttling and admission control. The talk detailed how the database manages different workload types—batch and interactive workloads—by either delaying client responses or failing new requests when necessary. For tables with materialized views, ScyllaDB 3.0 introduces advanced flow control that includes a linear control function to adjust client delays, ensuring that the system's memory and processing resources are not exceeded. These mechanisms allow ScyllaDB to operate efficiently without requiring manual configuration from users, supporting autonomous database management. The blog post also highlighted a simulation tool used to demonstrate these flow-control algorithms, emphasizing the goal of achieving worry-free ingestion by automatically adjusting to write loads.