May 2024 Summaries

At Railway, they prioritize leverage over management, focusing on building a high-signal, low-noise organization where employees are given autonomy and freedom to work on projects that align with the company's goals. They hire for obsolescence, seeking individuals who will challenge themselves and find innovative solutions, rather than redundancy. The company operates on a "pull-based" model, pushing information daily through channels like project channels and #progress-pics, reducing managerial overhead and allowing employees to focus on coaching, collaborating, and achieving greatness. With a culture of treating people like adults, Railway encourages direct feedback, perpetual learning, and work-life balance, where employees can choose when they're "on" or not. By valuing autonomy, innovation, and personal growth, Railway aims to create an inspiring environment where great things happen.

We recently experienced an outage on our platform that affected 1% of our compute nodes and less than 1% of our workloads, primarily impacting the Asia-Southeast cluster. The incident began at approximately 03:39 UTC when internal monitoring alerted to exhausted compute capacity in the region, followed by low-disk capacity alarms. Investigation revealed that all instances were experiencing high memory usage, utilizing swap memory to cover burst capacity requirements. A partial outage was declared, and mitigations included deploying a new compute node and manually culling workloads from existing nodes. However, one instance remained unresponsive due to being irrevocably soft locked, making further debugging efforts difficult. The root cause was identified as a self-service customer's workload exceeding the capacity of the region, causing memory spikes that triggered kernel OOM reaper actions. To mitigate similar incidents, Railway plans to adjust kernel OOM reaper configuration, implement workloads prioritization, and tune deployment algorithms to account for historical metrics and adjust compute node capacity in Asia-Southeast to decrease likelihood of saturation.

Railway Station is a public project that showcases the production applications of Railway, which includes its Blog, Docs, Forum, and more, serving hundreds of thousands of users daily. The idea behind Railway Station was to demonstrate how Railway can be used for complex, production workloads by consolidating services into a single project and using environments and environment sync to manage code and infrastructure changes. By doing so, the team learned valuable lessons about managing projects, organizing services, and improving visibility controls, which have been translated into feature revisions and enhancements. The project is now public and open-source, inviting users to explore and witness the growth of Railway's operations.