Everything as Code for Data Infrastructure with Airbyte and Kestra Terraform Providers

Everything as Code (EaC) is an approach that extends to managing not only software but also infrastructure and configuration in code. It involves using a Git workflow and code review process for changes to resources, rather than manual deployment. This post explores how EaC can be applied to data infrastructure, offering benefits such as easier environment reproduction, auditability, collaboration via pull requests, and automation of manual processes. The use of Airbyte Terraform provider is highlighted, along with its integration with Kestra orchestration for managing data ingestion, transformation, and scheduling. By adopting the EaC philosophy, data operations can become more resilient through the adoption of software engineering and DevOps best practices.