Better “Goodput” Performance through C++ Exception Handling

Piotr Dulikowski discusses the unexpected performance issues encountered while implementing per-partition query rate limiting in ScyllaDB, specifically focusing on C++ exception handling. Despite initial attempts to manage overloads by rejecting excess queries to maintain stable goodput, the handling of exceptions, particularly in large volumes, became a bottleneck due to the CPU-intensive nature of rejected operations. This issue was compounded by the use of Seastar, a C++ framework, which, while adept at managing asynchronous tasks, struggled with exception handling during high-load scenarios. To address these issues, the team explored two approaches: using boost::result to return errors instead of throwing exceptions, and implementing a custom utility for inspecting std::exception_ptr without rethrowing, thereby reducing the overhead of exception handling. Benchmarks comparing ScyllaDB versions 5.0.3 and 5.1.0-rc1 demonstrated that these optimizations significantly improved throughput and goodput, allowing the updated version to handle higher write rates with fewer timeouts, thus enhancing overall performance under stress.