Chasing that 32 ms tail latency in ClickHouse

ClickHouse engineers investigated an extra 32 ms of tail latency affecting simple Native TCP SELECT queries after connections had been idle for about two seconds across a 32 ms network path. The delay occurred because Linux reset TCP’s congestion window to its initial size after inactivity, while ClickHouse’s use of TCP_NODELAY caused separate small writes for result metadata such as ProfileEvents, logs, progress updates, and EndOfStream packets; together, these packets exceeded the reduced congestion window and required an additional client acknowledgement round trip. Using Linux network namespaces, traffic control, and a reused connection, the team reproduced the issue deterministically, observing results arrive after roughly one RTT but EndOfStream after a second RTT. Although configuration changes such as increasing the initial congestion window, disabling slow start after idle, or reducing profiling packets could mitigate the problem, the implemented ClickHouse fix batches suitable end-of-query metadata at the application layer while preserving timely progress messages. Packet captures showed that this nearly halved transmitted TCP packets, and the measured tail latency fell from about 32 ms to roughly 1 ms.

No tracked trend matches for this post yet.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.