Three bugs in the Go MySQL Driver
Blog post from GitHub
GitHub has been transitioning parts of its Rails monolith to Go, focusing on performance-critical components, such as the new authorization service "authzd," which interacts directly with production MySQL databases. This transition has uncovered challenges, including a persistent "Invalid connection" error within the Go MySQL driver, attributed to the aggressive idle timeout settings on GitHub's MySQL servers. The error was resolved by implementing a connection health check using non-blocking reads, which reduced overhead and improved service availability. Additionally, GitHub addressed a context handling issue in the Go MySQL driver that caused request timeouts to be ignored, necessitating a significant refactor to ensure context awareness throughout the connection lifecycle. A critical data race issue was also identified, related to the driver returning borrowed memory during SQL queries, which could lead to corrupted data when contexts were canceled mid-query. The solution involved implementing a double buffering technique to prevent data corruption without significant performance regressions. These improvements, now part of the Go MySQL driver's 1.5.0 release, highlight the complexities of deploying a new programming language at scale and GitHub's commitment to enhancing the Go ecosystem.