September 2026 Summaries

Mergify investigated persistent event-processing delays caused by a hash-based assignment scheme that placed organizations with widely different workloads across single-threaded Python worker processes without considering their actual cost, leaving one process saturated while others idled. Container-level CPU metrics obscured the problem because each container had two vCPUs but its asyncio worker could use only one, while a scheduling-latency unit error further understated saturation. Adding processes merely reshuffled organizations and did not reliably improve balance, so the company built a measured-load balancer that records total process CPU, apportions it among organizations using their relative busy time, stores explicit placements in Postgres, and moves a limited number of organizations based on high- and low-water thresholds. Development exposed two faulty load models: summing per-organization peaks overstated concurrent demand, while ranking processes by the peak of CPU measurements capped at one core made all busy processes appear equally saturated and incorrectly triggered capacity alarms. Replacing peak-based estimates with an exponentially weighted one-hour mean allowed the balancer to make 12 moves, reduce the gap between the hottest and coldest processes, eliminate repeated blocked-capacity alerts, and stop the stuck-event monitor without reported customer disruption.

A four-month CI performance issue was traced to GitHub Actions cache entries silently failing to restore because cache addresses depend not only on the visible key but also on an internal version incorporating the runner’s compression tool. GitHub-hosted cache-writing jobs used zstd while self-hosted test jobs lacked it and used gzip, causing identical cache keys to resolve to incompatible entries; restores reported ordinary “Cache not found” messages and exited successfully. Without the cached pytest duration data, tests were divided by count rather than expected runtime, intermittently leaving some shards far slower and causing timeout failures. The team resolved the mismatch by installing zstd on self-hosted runners and moving cache-writing jobs onto the same runner fleet as readers, while noting that enabling fail-on-cache-miss would have exposed the issue immediately. Early results showed no further timeout-killed shards, illustrating that successful CI checks do not necessarily confirm that caches are functioning as intended.

A security engineering team replaced an unenforced device-fingerprinting system, which relied on request headers and GeoIP data but could neither reliably identify attackers nor avoid disrupting mobile and VPN users, with browser-generated proof-of-possession keys that bind sessions to non-extractable WebCrypto credentials. After login, the browser registers an ECDSA P-256 public key with the server and signs authenticated requests, preventing a stolen cookie alone from being used elsewhere, although copied browser profiles and live cross-site scripting attacks remain significant limitations. Implementing the system exposed practical issues including WebCrypto and Python signature-format differences, incomplete request coverage, replay risks from a five-minute timestamp window, OAuth route exemptions, CSRF-like logout vulnerabilities, and a path-handling bug that initially caused all proofs to fail. The team used account-level counting mode before enforcement to limit rollout risk, removed obsolete fingerprinting dependencies and a 63 MB GeoIP database, and added protections for sensitive navigation routes using Fetch Metadata headers. The author concludes that the custom mechanism resembles the DPoP standard and should eventually be replaced where possible by Chrome’s Device Bound Session Credentials, which uses hardware-backed keys and short-lived cookie refreshes without application-level signing code, though its current lack of Firefox and Safari support requires a transitional approach.

A dashboard exposed a brief cross-account display issue because its React Query cache used one constant sessionStorage key for every login within the same browser tab, allowing a newly signed-in user to see a previous user’s cached profile and organizations until fresh network data arrived. The problem became visible after a resilience improvement allowed the application to render available cached data during GitHub outages rather than remain on a loading screen. Instead of attempting to infer sign-outs from unreliable 401 responses, the team scoped cache keys to a value generated or assigned uniquely at login, such as a request-signing session identifier, so each login reads a separate cache entry. This approach does not remove old data or protect against scripts already running on the same origin, and impersonation flows require explicit cache clearing because they intentionally retain a shared server session. Implementing the solution also introduced asynchronous IndexedDB timing risks, storage-eviction edge cases, and failure handling that expanded the initial small change substantially. The account emphasizes testing isolation by deliberately disabling the scoping mechanism and recommends that applications persisting browser caches inspect whether their storage keys are tied to the data owner rather than relying on detecting when cached data becomes stale.