March 2023 Summaries

Datafold announced that it is no longer supporting the open-source data-diff package as of May 17, 2024, and suggests a free trial of Datafold Cloud for in-database and cross-database diffing. The blog highlights the importance of using data diffs to catch unexpected changes in dbt models that standard dbt tests might miss, such as duplicated strings in a column. It introduces two open-source tools for running data diffs: dbt Labs' audit_helper and Datafold's data-diff. The audit_helper package includes macros like compare_all_columns, which can compare values across all columns of dbt models to identify differences, such as conflicting values, nulls, and missing primary keys between development and production versions of a table. By toggling the 'summarize' flag, users can either get a summary of differences or a detailed breakdown, allowing for deeper analysis and integration with warehouse data for more comprehensive insights.

The blog post discusses the discontinuation of active support and development for the open-source data-diff package by Datafold as of May 17, 2024, while promoting the continuation of its functionalities through a free trial of Datafold Cloud. It highlights data-diff as a Python CLI tool that facilitates efficient table comparisons within and across relational databases, data warehouses, and dbt models. The post details a tutorial demonstrating how to use data-diff reports for automatically testing dbt code changes in the Snowflake environment. It guides users through setting up a demo dbt project, configuring environments, and running data-diff to analyze the impact of SQL code changes. Prerequisites include having a Snowflake account and dbt-core installed, with steps provided for setting up a dbt project and configuring data-diff for Snowflake, including tagging columns with primary keys for efficient data comparison.

Data quality issues can disrupt key business operations, often surfacing in broken dashboards and inaccurate KPIs, but the data transformation tool dbt offers built-in testing to mitigate such problems. dbt-expectations, inspired by the Great Expectations library, provides a package of tests that are easier to set up and run faster than their counterparts, as they operate directly within a database. These tests, written in YAML templates with SQL, Jinja, and dbt macros, can be applied to various components of a dbt project, including sources, models, columns, and seeds, to address issues like incorrect data types, stale data, missing data, and non-unique or duplicate values. The package's tests surpass dbt's generic options, such as not_null and unique, by offering more granular checks and the ability to add row conditions. dbt-expectations is particularly useful for verifying column types, ensuring data freshness, and preventing missing data and duplicates, thus providing a comprehensive solution for maintaining data integrity and reliability in analytics workflows.

Data quality issues significantly impact business decision-making by affecting the accuracy, completeness, timeliness, or consistency of analytical data. These issues can arise at any stage of the data pipeline, including during ingestion, storage, transformation, orchestration, or consumption. The text categorizes data quality issues into production-specific and development-specific types, each with distinct causes and implications. Production-specific issues often stem from changes in third-party data sources or infrastructure failures, while development-specific issues are usually due to incorrect technology implementation, misunderstood requirements, or unaccounted downstream dependencies. Solutions for mitigating these issues include using data observability tools, such as Metaplane and Monte Carlo, which help detect anomalies in production environments. These tools are more effective than traditional infrastructure monitoring tools and can inform each other to enhance data quality. The importance of proactive detection to avoid costly errors or risks in production is emphasized, highlighting that many data issues are essentially bugs in data-processing software.