June 2024 Summaries

Managing static data in a data warehouse can be streamlined with dbt seeds, part of the Data Build Tool (dbt) framework, which modernizes traditional methods by integrating them into broader data infrastructure. dbt seeds are typically small, static CSV files that are easy to create, edit, and version control, allowing data teams to manage static data in the same way they handle code. This method enhances simplicity and consistency in data operations, as dbt seeds can be loaded into a warehouse during a dbt run and then used like any other table for joining with transformed data, modeling, or analysis. Unlike larger, frequently changing datasets that require custom ETL pipelines or tools like Fivetran and Airbyte, static data managed with dbt seeds benefits from a standardized approach, ensuring consistency and transparency through version control, pull requests, and peer reviews.

Datafold is now available for purchase on Google Cloud Marketplace, allowing data teams using Google Cloud Platform (GCP) to leverage pre-committed spend to acquire Datafold and enhance their data quality processes. This availability enables data teams to automate testing during the CI/CD process, reconcile data across databases, and monitor data effectively, addressing the growing need to scale, govern, and test data in response to increasing business demands. For teams evaluating data quality tools and facing budget constraints, using pre-committed Google Cloud spend presents a viable solution. By purchasing Datafold through the marketplace, teams can meet their committed spend levels while accessing a proactive data quality testing platform.

Navigating a data ecosystem can be challenging, but data catalogs provide an effective solution by organizing and managing data resources, much like a library. They allow users to quickly locate and utilize the necessary data, enhancing both data quality and accessibility. Beyond merely listing data locations, data catalogs offer metadata management, data profiling, and integration with data governance tools to ensure data integrity and compliance with regulations like GDPR or HIPAA. They also feature version control and collaboration tools, which improve data accuracy and facilitate teamwork. Advanced data catalogs employ machine learning for automated data discovery and classification, making data management more efficient and reducing human error. These tools are crucial for managing data quality dimensions and ensuring robust security and governance across organizations, offering varying levels of access depending on compliance needs.

Datafold's new No-Code CI integration offers a streamlined approach to incorporating data diffing into code review processes, enabling teams to proactively address data quality issues before they reach production. By connecting a git repository with data transformation code, users can automatically test and compare different versions of tables, such as staging versus production, directly within pull request comments. This integration supports various data transformation and orchestration tools and enhances the transparency of code changes on data, ensuring consistent testing practices across all code changes. Data engineers can leverage this tool to understand the impact of their code on data, facilitating improved governance, streamlined PR reviews, and accelerated data development. The integration also provides visibility into downstream impacts through its compatibility with BI tools like Tableau and Looker, and users can easily set it up by selecting the No-Code option and specifying the tables to be compared.

Datafold addresses the challenge of maintaining data quality in complex data pipelines by catching unintended changes to immutable data before deployment to production. While data engineers often focus on optimizing pipelines for performance, the accuracy of data across these systems is frequently overlooked due to the complexity of validation methods, such as dbt tests, unit tests, or manual SQL queries. Immutable data, which should remain constant over time, can change due to coding errors, data integration failures, data transformation errors, and data migration issues. Datafold proposes a simple solution to proactively detect these changes, which are not as uncommon as assumed, ensuring the integrity and reliability of data pipelines.