The Mysterious Gotcha of gRPC Stream Performance

Ably, a Platform as a Service (PaaS) focused on real-time messaging, encountered unexpected performance issues with a gRPC streaming server in its message-processing pipeline, particularly when tested under high throughput conditions. Despite gRPC's high-performance design and widespread use for connecting microservices, Ably's tests revealed unexpectedly high CPU usage, primarily due to system calls and scheduling goroutines, with performance hindered by the inefficiency of TCP packet handling. The investigation revealed that the observed inefficiencies were largely due to the timing of message publishing in the system under test (SUT), which differed significantly from a benchmark setup that packed more data frames into fewer TCP packets. This discrepancy highlighted gRPC's complexity and its challenges at high message rates, prompting Ably to consider ways to optimize network performance without re-architecting the system, while reinforcing the importance of understanding the real costs of data movement in a high-bandwidth environment like AWS. Ably remains committed to leveraging gRPC's benefits while being mindful of its limitations, continuing to explore solutions for more efficient real-time data exchange.