October 2023 Summaries

Airbyte has transformed its CI/CD process by adopting Dagger, an open-source orchestrator of workflows. The company's extensive ecosystem of 350+ containerized connectors requires consistent and reliable testing, making CI a significant challenge. Before Dagger, Airbyte's CI consisted of GitHub Actions YAML, shell scripts, and Gradle scripts, which were slow to execute, difficult to maintain, and led to high server costs. After transitioning to Dagger in February 2023, the team employed remote engines on a Kubernetes cluster leveraging Dagger Cloud for caching, optimization, and scalability. This transformation has resulted in faster local debugging, better maintainability, quicker pipeline execution, and significant cost savings of up to 75%. The collaboration between Airbyte and Dagger teams has been fruitful, with improved trust and resolution of issues. With Dagger Cloud, the team plans to continue building out their CI infrastructure, adopting other teams, and increasing observability and control over caching and parallelization configuration.

Dagger 0.9 introduces significant improvements to its services API, enabling Host-to-Container (H2C) and Container-to-Host (C2H) networking, as well as explicit service lifecycle management. This allows Dagger pipelines to expose services running inside them to the host for fine-grained control over network resources. The new API also expands Dagger's capabilities towards Application Delivery as Code, unlocking a vast number of new use cases such as spinning up microservices locally in Dagger, using AI language models/services on the host, and connecting to remote services via VPNs. With the updated SDKs, developers can now leverage these features with bug fixes and improvements. The latest release brings bespoke service lifecycle management, allowing users to start and stop services explicitly, and provides code samples for Go, Node.js, and Python.

In this blog post, Kasper Hermansen from Lunar shares the story of how his team transitioned from shell scripts and Docker to Dagger, a platform engineering tool that enables engineering principles across CI/CD pipelines. The transition improved the reliability and consistency of their software delivery platform, allowing developers to work more efficiently and reducing build failures. With Dagger, Kasper's team can now test and package their tooling with ease, collaborate on improvements, and deliver software faster. Although some challenges remain, the benefits are clear: happier developers, more efficient software delivery, and a better experience for internal development teams. As Kasper continues to experiment with Dagger's features, he is eager to explore its potential in ephemeral local development environments and extensions.