April 2025 Summaries

The Railway team recently experienced a Major Outage that affected their API backend, impacting users who relied on the dashboard, CLI, and Public GraphQL API. The outage was caused by configuration changes made to the API backend's infrastructure, which introduced issues with service mesh internal proxy-ing and unoptimized SQL queries. To address this, the team rolled back changes, fully reverted them, and then fully removed the service mesh from their API backend. They have implemented new strategies for rolling out configuration changes, including staggering deployments and having rollback plans in place. The incident highlighted the importance of providing a best-in-class cloud experience and has led to a renewed commitment by Railway to eliminate similar issues in the future.

We recently experienced a networking outage on Railway that affected public and private networking across multiple regions, with the Asia region being impacted the most. The outage was caused by a maintenance issue on Google Cloud SQL's Postgres service, which terminated all connections to our network control plane, leading to routing information loss and subsequent crashes in some instances. The root cause was traced back to our global control plane design, where we should have had regional control planes instead of a single global one to contain the blast radius of failures. To fix this, Railway is prioritizing an upgrade to regional control planes and implementing measures such as persistent caching, improved startup times, and enhanced Google Cloud SQL configuration to prevent similar incidents in the future. The outage was resolved after 10 minutes, but not before causing instability for users visiting Railway-hosted domains or communicating over our private network.