Incident Report: December 8th, 2025
Blog post from Railway
Railway experienced a significant outage on December 9th, 2025, disrupting its backend API, dashboard access, CLI operations, GitHub-based deployment processing, login, and API functionality, although running deployments remained unaffected. The incident was triggered by a database schema change involving a data migration, which encountered lock conflicts due to a long-running query on a heavily utilized table with approximately one billion rows. This led to backend replicas becoming unhealthy and failing due to exceeded connection limits managed by PgBouncer, removing them from service. Once the query completed and released its locks, the migration finished, allowing replicas to reconnect and stabilize the backend. To prevent similar occurrences, Railway plans to increase database connections, reconfigure PgBouncer, and address the long-running query to avoid future lock issues, emphasizing the importance of reliability to their users.