May 2024 Summaries

Data observability and data testing are crucial for maintaining high data quality, as manual inspection is insufficient to catch all potential errors in data, such as formatting issues, off-by-one errors, or time zone-related anomalies. Data validation, which involves verifying the quality and consistency of data at its source, is essential before integrating it into a broader data collection system. Automation is strongly recommended for this process, as it enhances efficiency, accuracy, and scalability while minimizing human error. Tools like Datafold and dbt are popular choices for automating data validation, offering features like cross-database data diffing and source-level data tests to ensure parity between source and target systems. Effective data validation extends beyond the initial transfer to encompass the integrity of data within production environments, where its accuracy and reliability are crucial for real-time applications and decision-making. Automation ultimately streamlines data validation, making it more efficient and reliable than manual methods.

The text explores the concepts of data integrity and data quality, emphasizing their importance in discussions about generative AI tools like ChatGPT. Data integrity involves maintaining data accuracy, consistency, reliability, and completeness throughout its lifecycle, ensuring it is stored in its original form and safeguarded through input validation, data constraints, transactions, and audit trails. In contrast, data quality focuses on the suitability of data for its intended downstream applications, encompassing dimensions such as accuracy, completeness, consistency, reliability, timeliness, uniqueness, usefulness, and differences. The text dispels common misconceptions, such as equating data integrity solely with data preservation or assuming data quality is merely about accuracy. Enforcing data integrity and quality involves continuous monitoring, validation at data collection points, and using tools like automated testing and anomaly detection to maintain accuracy and relevance. The distinction between the two lies in their purpose, with integrity ensuring data is trustworthy for critical decisions and quality ensuring it meets specific user and application needs.

As of May 17, 2024, Datafold has decided to cease active support and development for the open source data-diff tool to concentrate on enhancing their primary product, Datafold Cloud. Datafold was founded in 2020 to address critical aspects of data quality in a data engineer's workflow, such as data reconciliation and transformation code changes, through technology that compares datasets at any scale. Initially launched as a SaaS product, Datafold later introduced data-diff as an open source tool to increase accessibility within the community. However, the increasing demand for Datafold Cloud and the need to maintain two separate products with overlapping features have prompted the decision to focus exclusively on the SaaS offering. The company believes concentrating resources on Datafold Cloud will ultimately benefit its customers and the company, despite ending the open source project. Datafold plans to continue innovating data quality testing, having recently implemented improvements like sampling, real-time results, and enhanced database support, as they aim to cover the entire data quality lifecycle.

Data lineage is crucial for understanding the transformation of raw data into usable models within data pipelines, often visualized as directed acyclic graphs (DAGs) with cataloging and metadata for comprehensive insight. dbt, both as an open-source tool and a SaaS platform, is widely used by data engineers for data transformation and takes advantage of its native data lineage features for data discovery and impact analysis. Lineage tools are essential for identifying downstream impacts to prevent deployment issues, enable root cause analysis during incidents, and assist in data discovery by tracing upstream dependencies for decision-making. While dbt's native lineage offers valuable insights, enhancing it with robust column-level lineage and proactive impact analysis testing can further support data teams in ensuring data quality and reliability.

In the realm of data analytics, dbt Exposures play a critical role in mapping the relationship between data models and the end-user data products, such as dashboards and reports, by extending dbt's native documentation capabilities. While dbt provides automatic table-level lineage and dbt Cloud offers column-level lineage, Exposures allow developers to document how data models are utilized by the business within the dbt directed acyclic graph. Defined in YAML files, Exposures help identify dependencies and ownership of data products, ensuring clarity in upstream and downstream data flow. Best practices for implementing dbt Exposures include starting with the most critical business assets, establishing team guidelines for consistent documentation, and keeping Exposures up-to-date to maintain trust and utility. By focusing on these practices, data teams can enhance the reliability and transparency of their data products, thereby facilitating better decision-making processes within organizations.

An effective testing strategy is essential for delivering high-quality data products, as it enhances team efficiency while reducing data quality issues. Implementing an effective data testing strategy with dbt involves several best practices, such as shifting testing to the left, establishing a foundation with generic dbt tests, and using unit testing for complex code logic. Additionally, data diffing is useful for identifying unknown unknowns, and it is crucial to test data during continuous integration (CI) while avoiding deploying failed pull requests. dbt testing is a proactive process to ensure data quality by validating data-processing code before deployment, preventing regressions from changes like SQL column alterations or code refactoring that could impact business metrics or downstream systems.

Many companies using dbt projects often rely on unit tests or data tests, but a robust data quality strategy requires a combination of both, along with data diffs, to address their differing purposes and limitations. dbt unit tests, introduced in dbt Core 1.8, validate model logic with predefined inputs and outputs, allowing for granular testing and isolation of the model code from source data validation. Unlike dbt data tests, which execute assertions on live, ever-changing data, unit tests offer faster execution and less noise, making them ideal for complex SQL logic and critical business scenarios. While dbt data tests, known for their simplicity, can be computationally expensive and noisy, unit tests provide a more stable testing environment during the development phase, especially useful for edge cases or retrospective bug fixes. Integrating dbt unit tests into the CI process is recommended to assess code changes during reviews, but they are not suitable for production due to increased execution time without added benefit.

Opening a pull request to modify dbt models can be stressful, as even successful CI pipeline runs and passing dbt tests may not reveal all potential data changes introduced by the modifications. While dbt tests are designed to catch certain data quality issues, they may not cover all scenarios, potentially leading to broken dashboards and malfunctioning pipelines if unexpected data changes occur. Datafold is introduced as a complementary tool to dbt tests, providing additional test coverage by identifying value-level differences between staging and production data. This helps prevent issues such as errors in data values, distribution shifts, and missing table sections that dbt tests might miss. Therefore, integrating Datafold into CI pipelines alongside dbt tests is recommended to ensure the accuracy, completeness, and consistency of data, ultimately safeguarding data quality when deploying code changes to production.