Company
Date Published
Author
-
Word count
4220
Language
English
Hacker News points
None

Summary

Airbyte has transformed its CI/CD process by adopting Dagger, an open-source orchestrator of workflows. The company's extensive ecosystem of 350+ containerized connectors requires consistent and reliable testing, making CI a significant challenge. Before Dagger, Airbyte's CI consisted of GitHub Actions YAML, shell scripts, and Gradle scripts, which were slow to execute, difficult to maintain, and led to high server costs. After transitioning to Dagger in February 2023, the team employed remote engines on a Kubernetes cluster leveraging Dagger Cloud for caching, optimization, and scalability. This transformation has resulted in faster local debugging, better maintainability, quicker pipeline execution, and significant cost savings of up to 75%. The collaboration between Airbyte and Dagger teams has been fruitful, with improved trust and resolution of issues. With Dagger Cloud, the team plans to continue building out their CI infrastructure, adopting other teams, and increasing observability and control over caching and parallelization configuration.