December 2023 Summaries

The Railway engineering team experienced a production outage from 21:45 UTC to 23:24 UTC due to issues with Google's Metadata server, which was upgraded as part of the rollout of GKE v1.25. The upgrade caused significant delays in encrypt and decrypt requests, affecting the ability of users to deploy new workloads and update environment variables. The team quickly identified and addressed the issue by rolling out a newer version of GKE that was not affected by the known issue, and restored all services to normal operation by 23:24 UTC. The incident highlighted the importance of careful planning and monitoring during significant infrastructure changes, and led to several takeaways for improving internal coordination, messaging, re-shoring, and incident management.

Not everything is Google's fault, but most things are. The author of the text, Railway, has had a series of issues with Google Cloud Platform (GCP) products, including softlocks in rare cases under resource pressure, arbitrary quota changes, and unreliable customer support. Despite escalating these issues to GCP, Railway found that they were unable to get satisfactory responses or resolutions. As a result, Railway built their own networking stack and registry product to avoid these issues. However, after experiencing another issue with Google Cloud, Railway decided to sunset all Google Cloud services and move to bare metal instances in 2024. The latest incident involved a box dropping offline due to a Google restart, but despite manual failover efforts, the issue persisted, leading Railway to conclude that there is a potentially fatal interaction in userspace-to-kernel memory transfer on GCP guests that causes softlocks in rare cases under resource pressure.

The Railway platform experienced a 30% outage on its US-West compute infrastructure, causing workloads to become unreachable for up to an hour. The root cause was attributed to a newly deployed metrics collection agent that triggered a CPU core soft lock due to excessive memory transfers between kernel and userspace. This issue was only detected after the deployment had begun and affected a subset of machines randomly throughout the fleet, with no discernible pattern or warning signs. The team manually restarted affected hosts, and by 20:23 UTC, all hosts were back online, with total confirmation from customers that workloads were restored. The incident highlighted the need for improvement in internal deployment tooling, building resilience, removing "black box" systems, better system checks, and mandating staged rollouts for all services.