This SRE attempted to roll out an HAProxy config change. You won't believe what happened next...

Originally published on the GitLab Unfiltered blog and republished in February 2021, this blog post discusses an issue encountered during the rollout of the gitlab-pages PROXYv2 project involving HAProxy's server-state-file directive, which causes the persistence of old port numbers across restarts. This resulted in failed connections as HAProxy attempted to send traffic to outdated ports despite configuration changes in haproxy.cfg. The problem was traced to HAProxy's state file, which retained previous server states, including port numbers, and overrode new configurations upon reloading. A workaround was devised by changing the backend server names in the configuration file, which prevented HAProxy from matching them with the old state, thereby allowing the updated port numbers to take effect. The blog also delves into troubleshooting steps, including traffic capture analysis and the use of tools like tcpdump and opensnoop, and explains the mystery behind unexpected "QUIT" messages found in traffic captures, which were identified as part of the PROXY protocol's version 2 binary header format.

No tracked trend matches for this post yet.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.