Why So Slow? Using Profilers to Pinpoint the Reasons of Performance Degradation

When implementing a new feature or switching libraries, application performance can suffer, as experienced by the team at Memgraph when transitioning their Bolt server to use Boost.Asio. Initially, their handwritten Bolt server faced performance issues after they shifted to support WebSocket, a decision driven by the increasing popularity of WebSocket and the introduction of a monitoring server via WebSocket in Memgraph 2.2. Despite successfully passing tests, performance benchmarks revealed significant degradation, prompting the team to use profilers like Callgrind and gperftools to diagnose the issue. While Callgrind provided detailed call statistics, it failed to pinpoint the slowdown, leading them to gperftools, which revealed that the send function was unusually slow due to the absence of the MSG_MORE flag, causing every call to send an actual package over the network. Correcting this oversight by utilizing boost::asio::basic_stream_socket::send instead of boost::asio::write restored performance closer to original levels, highlighting the importance of understanding both networking intricacies and profiling tools. Through this process, the team learned valuable lessons about package handling, the significance of correctly instrumenting POSIX sockets, and the complementary use of different profilers to diagnose performance issues effectively.