How we cut our CI pipeline execution time in half

Last year, the Tinybird team faced challenges with their continuous integration (CI) pipeline, which had become unreliable and inefficient due to the rapid growth in their team, codebase, and test suite. To address these issues, they paused all production changes for a week to focus on fixing the CI system. They initially implemented parallelization and auto-scaling techniques to improve CI performance but soon realized these changes did not address underlying problems. By utilizing their own product, a real-time analytics platform, they collected and analyzed CI telemetry data, leading to insights that helped optimize test execution order, redistribute tests across workers, and tailor infrastructure resources to specific job requirements. These optimizations resulted in a 60% reduction in CI pipeline execution time and the elimination of random job failures. The process underscored the importance of data-driven decision-making, observability, and collaboration in engineering. Despite these improvements, the team continues to seek further optimizations and plans to share their solutions with the wider development community.