hash(id) % 8 Pinned One Python Process at 98% While Its Neighbors Idled

Mergify investigated persistent event-processing delays caused by a hash-based assignment scheme that placed organizations with widely different workloads across single-threaded Python worker processes without considering their actual cost, leaving one process saturated while others idled. Container-level CPU metrics obscured the problem because each container had two vCPUs but its asyncio worker could use only one, while a scheduling-latency unit error further understated saturation. Adding processes merely reshuffled organizations and did not reliably improve balance, so the company built a measured-load balancer that records total process CPU, apportions it among organizations using their relative busy time, stores explicit placements in Postgres, and moves a limited number of organizations based on high- and low-water thresholds. Development exposed two faulty load models: summing per-organization peaks overstated concurrent demand, while ranking processes by the peak of CPU measurements capped at one core made all busy processes appear equally saturated and incorrectly triggered capacity alarms. Replacing peak-based estimates with an exponentially weighted one-hour mean allowed the balancer to make 12 moves, reduce the gap between the hottest and coldest processes, eliminate repeated blocked-capacity alerts, and stop the stuck-event monitor without reported customer disruption.

No tracked trend matches for this post yet.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.