May 2023 Summaries

Datafold has announced its availability on the AWS Marketplace, enabling data teams to purchase its data quality testing tools for dbt code using their pre-committed AWS spend, thus enhancing accessibility. This move allows companies to leverage their existing AWS Enterprise Discount Program (EDP) commitments to procure third-party software like Datafold without needing additional budget or formal approval processes. This is particularly beneficial for companies with budget constraints, as it facilitates the acquisition of necessary data tools through existing cloud investment. More information can be found on Datafold's website or by contacting their team.

The text provides an overview of a meetup featuring talks from several experts in the fields of analytics, data science, and data engineering. Emily Riederer of Capital One discussed the importance of latent communication within data stacks and the role of column-name contracts in providing real-time context, using dbt to create scalable and resilient data pipelines. Felix Kreitschmann and Jorrit Posor from FINN Auto focused on the balance between technical alignment and speed in organizational structures, exploring the tradeoffs of centralization versus decentralization. They also highlighted the benefits of automated continuous integration checks for maintaining code quality, validity, lineage, and policies.

At Datafold, the dbt project initially had a simple setup with a single GitHub Actions job running full builds daily and for every code change, but as contributions increased, this led to prolonged CI feedback loops and higher Snowflake costs. To address these issues, the team restructured their CI/CD processes, resulting in a 30% reduction in Snowflake expenses and improved productivity. They optimized by implementing discrete jobs for specific tasks, running only necessary models, and excluding unchanged views and static models. Additionally, they introduced Slack notifications for job statuses to ensure prompt awareness of any failures, streamlining the workflow and making it more efficient and cost-effective.