Designing a Userspace Disk I/O Scheduler for Modern Datastores: the ScyllaDB example (Part 1)

In datastores like ScyllaDB, a userspace Disk I/O Scheduler is crucial for managing competing disk I/O requests, ensuring that prioritization and fair allocation of resources are maintained, unlike traditional methods that rely on kernel-level scheduling. The Seastar framework, used by ScyllaDB, employs a thread-per-core design to enhance scalability and avoid iowait times, which can occur when requests exceed the processing capacity of lower layers in the stack, such as the filesystem or disk array. By managing I/O requests in userspace, ScyllaDB can provide rich processing capabilities, including request prioritization and cancellation, ultimately reducing latency and improving performance. Experiments demonstrate that using an I/O Scheduler can significantly lower latency at the 99th and 99.9th percentiles, even in noisy environments like Amazon EC2. The upcoming second part of the article will delve into the design of the I/O Scheduler and its role in ensuring fairness among multiple users.