March 2023 Summaries

At Dagger, CI/CD is a crucial aspect that they think about extensively. They release as often as necessary in small batches and prioritize fixing issues within days. Larger features are released behind an experimental environment variable flag to gather real-world feedback from users. Five different artifacts - Dagger Engine, CLI, Go SDK, Python SDK, and Node.js SDK - are released, all originating from the monorepo github.com/dagger/dagger. The release process involves GitHub Actions workflows that trigger on specific git tags, publishing artifacts to various registries like registry.dagger.io. Documentation is a cross-cutting artifact closely coupled with the other releases, built on Netlify for two separate websites. Dagger uses its own SDKs to publish itself, showcasing "first-class dogfooding." To improve their release process, they are considering announcing new SDK releases in Discord and configuring a release bot, automatically publishing documentation when new SDK releases go out, and improving their release notes to be friendlier and more readable.

In this blog post, the author shares the story of Andy Gonzalez, an SRE team leader who used Dagger to speed up his production AWS Lambda deployments from a few hours to just a few minutes, achieving a 93% reduction in processing time. Previously, deploying Lambdas was a manual and time-consuming process that took between 1-3 hours every time, with challenges including complex YAML files and a lack of automation. Andy replaced this process with Dagger, which enabled him to deploy Lambdas in parallel using matrix jobs, significantly speeding up the deployment process. The new approach also encapsulated deployment business logic within Dagger pipelines, making it easier to test and maintain. Going forward, Andy plans to combine his existing CircleCI pipeline with Dagger and migrate the CircleCI pipeline to Dagger, expecting this will make the deployment process even more efficient.

Today's release of Dagger 0.4.0 introduces service containers, which enable the run of network services as ephemeral containers within a Dagger pipeline. This feature is particularly useful for running test databases, end-to-end integration tests, and sidecar services. Service containers are started just-in-time, de-duplicated, and stopped when no longer needed, with built-in features such as health checking and hostname aliasing. The release also includes secrets scrubbing, which automatically removes sensitive data from logs and output streams. Additionally, new versions of Dagger's SDKs have been released with support for the new features, along with various bug fixes and improvements.