Async Rust in Practice: Performance, Pitfalls, Profiling

The ScyllaDB Rust Driver, initially developed during an internal hackathon, has undergone significant advancements since its inception, but encountered unexpected performance issues compared to the cassandra-cpp driver, particularly in local testing environments. These issues were traced back to the lack of buffering in the driver's read and write operations, leading to excessive CPU usage due to frequent syscalls. By implementing buffering with Tokio's BufReader and BufWriter, the driver significantly reduced syscall overhead. A subsequent challenge involved the use of Rust's FuturesUnordered, which caused quadratic execution time due to its interaction with Tokio’s cooperative scheduling. This was resolved by limiting the number of futures iterated in a single poll, balancing cooperative scheduling benefits and maintaining low latencies. The experience underscores the ongoing evolution in Rust’s async programming, highlighting the importance of profiling and community contributions in optimizing performance.