April 2023 Summaries

Using dbt Cloud in Continuous Integration (CI) and implementing Slim CI can enhance efficiency and reliability in data model building and testing. These practices allow users to catch potential production run failures early by executing dbt Cloud Pull Request (PR) Jobs in CI, ensuring all dbt models build successfully and passing tests before reaching production. Slim CI optimizes this process by selectively building and testing only the dbt models edited in the PR or those downstream of the edited models, saving time and cost. Setting up a dbt Cloud PR Job involves creating a job that mirrors the production environment and triggers a `dbt build` command whenever commits are pushed to a PR, writing the results to a dedicated section of the cloud data warehouse. This approach verifies the functionality of the current code version and provides the opportunity to query and investigate the PR data, though it may require significant resources if the entire project is rebuilt frequently.

Datafold has announced it will no longer support or develop its open-source data-diff tool as of May 17, 2024, instead encouraging users to try Datafold Cloud for database diffing. The company was founded to help data engineers ship faster without sacrificing quality, addressing challenges like the lengthy testing and validation processes in data engineering. A pivotal experience at Lyft highlighted the need for better tools to prevent errors from affecting downstream data pipelines. Datafold emphasizes the importance of data diff and data lineage technologies to provide visibility into the impact of code changes on datasets and downstream applications, advocating for deployment testing during the development process. Over the past three years, Datafold's automated pre-production testing helped data teams identify regressions and align with data consumers, with many users interested in early-stage testing, reflecting the shift-left trend in software engineering. Although Datafold open-sourced the data-diff tool to support efficient data comparison, they recognize that maintaining high data quality requires a comprehensive workflow beyond just data comparison.

The concept of data freshness is crucial in ensuring data quality by measuring how up-to-date the data is, which is vital for businesses relying on near real-time data to make informed decisions. Testing data sources for freshness, especially at the ingestion phase, helps detect issues early and assists in debugging by pinpointing whether problems originate at the source or during data transformation. Tools like dbt provide commands to test data source freshness, using a 'loaded_at_field' to compare timestamps against expected update frequencies, allowing error or warning alerts based on set thresholds. Implementing these tests requires careful consideration of each data source's update patterns to avoid alert fatigue, and it is recommended to run freshness tests before executing any data models. The results of these tests provide valuable insights and are stored in a 'sources.json' file, helping track the history of data freshness and identify recurrent issues.

Testing data pipelines is a complex yet crucial process that aims to ensure changes made to the pipeline yield the desired effects without introducing regressions, though achieving absolute certainty remains elusive. The process involves setting up development and staging environments to inspect and share data changes, reviewing code modifications, running pipelines in different environments, and querying and writing tests for data at various layers of the modern data stack. This stack comprises storage, orchestration, integration, transformation, visualization, and activation layers, each demanding specific testing strategies. Tools like zero-copy clones in Snowflake and table clones in BigQuery facilitate testing with production data, while orchestrators like Airflow and Dagster offer various capabilities and challenges in managing tasks and testing setups. As the landscape evolves, the integration and testing of these diverse components continue to pose challenges, prompting ongoing improvements and innovations in the tools and practices used to enhance confidence in data pipeline modifications.