The case of the spiky file descriptors

In an exploration of a peculiar issue with file descriptors at Tailscale, the team discovered a sawtooth pattern in the file descriptor count after deploying stateless reverse proxies to reduce HTTP connections to the coordination server. Despite the pattern being benign, it coincided with the release of a new server build, prompting an investigation. The root cause was traced to the autocert package's handling of Let's Encrypt certificates, which was attempting to renew certificates for a domain now served by reverse proxies. This resulted from users hardcoding DNS entries, causing certificate requests to reach the coordination server instead of the proxies. The problem was exacerbated when previous certificates expired and a server deployment reset caches, necessitating certificate re-requests. By removing the domain from autocert's allowed list, the team resolved the issue, highlighting the complexities of handling diverse client configurations and reinforcing the importance of detailed investigation to ensure seamless service operations.