Our New Observability Platform for 100B Logs

This is a story about how a company, Railway, rebuilt its logging infrastructure to handle 1 billion new logs per day. The old system was struggling and they wanted a scalable solution that could support filtering and other advanced features. They discovered ClickHouse, a column-oriented database used for log aggregation and analytics, which seemed like an ideal fit. After testing and implementing ClickHouse, Railway was able to seamlessly integrate it into their existing logging architecture, allowing them to add new features like Log Explorer, which provides an environment-level view of all logs. The new system has improved performance, scalability, and speed, making it easier for the company to identify and fix problems quickly.