Accelerating dbt core CI/CD with GitHub actions: A step-by-step guide

At Datafold, the dbt project initially had a simple setup with a single GitHub Actions job running full builds daily and for every code change, but as contributions increased, this led to prolonged CI feedback loops and higher Snowflake costs. To address these issues, the team restructured their CI/CD processes, resulting in a 30% reduction in Snowflake expenses and improved productivity. They optimized by implementing discrete jobs for specific tasks, running only necessary models, and excluding unchanged views and static models. Additionally, they introduced Slack notifications for job statuses to ensure prompt awareness of any failures, streamlining the workflow and making it more efficient and cost-effective.