September 2022 Summaries

As the data world increasingly adopts software development concepts, the article explores the application of unit and integration testing in data contexts, highlighting tools like dbt and Great Expectations. Unit tests are designed to verify the basic functionality and handle edge cases of individual pieces of code, such as functions or classes, ensuring they behave as expected. dbt offers a straightforward framework for unit testing database tables, allowing users to specify conditions like uniqueness or non-null values with minimal configuration. Integration tests, on the other hand, ensure that different components of an application work together correctly, such as maintaining referential integrity between database tables. dbt facilitates these tests as well, enabling validation of foreign key relationships with concise YAML configurations. Great Expectations complements these capabilities by offering a Python API and an extensive library of tests, with a web UI in development, providing a robust alternative for data testing.

Monitoring source data freshness in dbt can be challenging due to the manual configuration of expected cadences, which often leads to alert fatigue from false alarms. To address this, the integration of Datafold alerts, which utilize machine learning-based forecasting models, allows for the dynamic configuration of thresholds that adapt to data seasonality and trends, reducing noise and improving response efficiency. By creating a dbt snapshot of the information schema and writing SQL queries to track table changes, users can leverage Datafold to detect anomalies in data freshness. This approach simplifies the alerting process, enabling alerts to be configured easily and delivered via Slack or email, although it is specifically applicable to materialized tables, as views lack row counts. The method is demonstrated using the Snowflake database but can be adapted for other databases.

Apache Airflow is a versatile platform used for writing, scheduling, and monitoring workflows, providing a centralized way to manage and visualize tasks within a data ecosystem. It supports a wide range of "Operators" for diverse functions beyond SQL scripts, such as making HTTP requests and running Python code, which highlights its role in orchestrating comprehensive data pipelines. The integration of Airflow with dbt allows for enhanced task dependency management, centralized control, and parameterization across systems, offering a holistic view and alert system for the entire data pipeline. A detailed demo is provided, guiding users through setting up a local environment using Docker to install Postgres, dbt, and Airflow, demonstrating how to configure and run a data workflow. The demo emphasizes the use of containers for simplicity and organization, illustrating the practical steps for creating and managing data workflows in a structured and accessible manner.

The text discusses the concept and importance of data contracts in modern data management, highlighting how they can mitigate data quality issues by formalizing interactions between different systems and teams. Drawing an analogy with software development, where well-defined interfaces allow for error detection at compile-time, the author argues that similar mechanisms are necessary for data systems to prevent disruptions caused by schema changes. The increasing complexity, integration, and real-time requirements of analytical data underscore the need for such contracts, as they help ensure consistency and reliability, much like APIs do for web services. By providing structured, predictable interfaces to data, data contracts can prevent errors and facilitate smoother transitions when changes occur, ultimately safeguarding against costly production errors and improving overall data quality.