January 2024 Summaries

The Railway team recently experienced a significant Distributed Denial of Service (DDoS) attack that lasted for approximately 2 hours and affected users worldwide, particularly those in Europe. The attack peaked at over 12 million requests per second, making it one of the largest DDoS attacks ever recorded on the platform. To mitigate the attack, Railway engineers worked tirelessly to isolate the hostile traffic, implement additional services to reduce the attack's impact, and rate limit incoming connections. Despite initial challenges, they were ultimately able to recover from the incident by inserting a new service in the critical pathway between their load balancer and edge network, which blunted the traffic enough to allow for further rate limiting. The team has since filed a ticket with their upstream provider to address the issue and is working on additional preventative measures to improve security and mitigate future attacks.

Support engineers at Railway are full-stack engineers who focus on building interfaces between humans and systems. They use a unique approach that prioritizes bringing structure to human interactions and efficient information transmission. The team uses the OSI model of support engineering, where every support outreach is an opportunity to improve the product. They treat users as customers and aim to repair cracks in the product and provide soft landings for users who fall through the cracks. The team has a rotating support function that includes non-technical staff and product area experts, which frees up engineers to build lasting product improvements. The team educates users through open-source documentation and builds forms to resolve user issues quickly. They also centralize communications and automate on-call handover notices to reduce context switching. The goal is to have 1 support engineer for every 1M total users and compound their abilities through a feedback control loop, ultimately aiming to 100x the product and organization.