How posix_spawn & close_range “fixed” cold starts

In November, an engineering team faced a regression in cold start latency due to changes made to improve stability and tenant density in their Neon Pageserver system. The issue arose from a decision to shut down idle walredo processes to conserve DRAM capacity, which inadvertently increased latency for cold starts and first queries after idling. Initially, the team suspected inefficient process creation using fork and exec instead of posix_spawn as the root cause, due to the need for a defense-in-depth strategy to close file descriptors before executing untrusted code. Upon further investigation, it was discovered that the real latency culprit was the inefficiency in closing file descriptors, which was resolved by moving the file descriptor closing logic to the walredo subprocess. This change allowed the use of posix_spawn, significantly reducing latency by preventing the fallback to a slower method of closing file descriptors. The experience highlighted the importance of dynamic tracing and proper monitoring to identify and resolve performance issues effectively.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.