How Slack Transformed Their CI With Tracing

Between 2017 and 2020, Slack experienced rapid growth, which led to significant challenges in its continuous integration (CI) system, particularly with flaky tests that undermined developer trust and slowed down deployment processes. Frank Chen, a Senior Staff Engineer at Slack, addressed these issues by implementing an observability-driven approach to trace and diagnose CI infrastructure problems, significantly reducing the flaky test rate from 50% to 5%. The solution, involving tools like SlackTrace and Honeycomb, allowed for real-time analysis of trace data, enabling quick identification and resolution of issues such as slow Git checkouts and system overloads. This transformation not only improved developer velocity and confidence but also fostered a collaborative effort among teams to maintain a robust CI environment, highlighting the critical role observability played in managing and optimizing Slack's complex infrastructure.