ScyllaDB’s New IO Scheduler

ScyllaDB's new IO Scheduler, introduced in the ScyllaDB Open Source 4.4 release, addresses the challenge of balancing disk input/output (IO) operations to optimize performance while minimizing latency. The new scheduler enhances the previous setup by allowing all shards to perform their own IO tasks independently, avoiding overburdening a single CPU and adhering to the principle of dispatching as much as the disk can handle. This is achieved through the introduction of IO groups, which facilitate fair capacity allocation among shards and prevent static disk partitioning that could limit performance. The new scheduler employs a "capacity rovers" algorithm, akin to TCP's congestion control, to manage request dispatching efficiently without the drawbacks of earlier methods. This improvement is particularly beneficial for systems with imbalanced workloads and opens up possibilities for additional features like IO cancellation. Users can test its performance using the io_tester tool in the Seastar repository, and the new scheduler is expected to deliver more uniform latency distributions compared to its predecessor.