January 2024 Summaries

Cloud data migrations can be categorized into two main strategies: "lift and shift" and "migrate and refactor," each presenting varying levels of difficulty. The "lift and shift" approach involves transferring data as-is from an old system to a new one, with success measured by consistency between the two systems. Meanwhile, the "migrate and refactor" strategy requires the adaptation of functions, emphasizing the need to validate that numerous functions continue to operate correctly. Successful data migration, regardless of the method, necessitates thorough testing to ensure the transition's efficacy, and understanding the best testing strategies is crucial for overcoming the challenges associated with these migrations.

Migrating data to the cloud presents numerous challenges, often revealing unexpected complexities, such as the significant differences between Oracle's and Snowflake's SQL, which complicate the process far beyond simple script writing. The tooling landscape for cloud data migration is crowded and confusing, as many tools offer similar-sounding features but differ in functionality, often requiring manual intervention and lacking proper documentation. Additionally, the market is heavily influenced by enterprise-focused providers, making useful information scarce and the search for effective tools difficult due to a pay-to-play environment. The goal is to simplify this process by offering guidance on the most useful tools for successful data migration.

Datafold has introduced new integrations for Bitbucket and Azure DevOps (ADO) within its Datafold Cloud service, enabling users to incorporate data diffing directly into their Continuous Integration (CI) processes. This integration allows users to gain enhanced visibility into code changes and streamline the pull request (PR) review process by automatically adding a Datafold CI comment to Bitbucket or ADO PRs. The comment provides insights into data differences between development and production models, potential changes in downstream tables, and alterations in data app assets like Looker and Tableau. Users can delve deeper into these data changes via a link to the Datafold Cloud app, facilitating detailed analysis and collaboration. By embedding data diffing into their CI workflows, data teams can improve data quality assurance, rapidly identify issues, and prevent flawed data from entering production pipelines. Datafold encourages interested teams to explore these features and offers resources such as a live demo and expert consultations to help integrate Datafold into existing data engineering stacks.

The integration of Datafold with MotherDuck using the open-source tool data-diff offers a solution for data engineers to verify the accuracy of their data pipelines by ensuring that replicated and transformed data matches expected outputs. This integration assists in validating the replication of data from sources like Postgres to databases such as Snowflake, Databricks, BigQuery, or MotherDuck, and in confirming that transformations made using dbt are correct before merging into production. Through the use of data-diff, data engineers can easily identify discrepancies and resolve them, ultimately fostering a reliable data environment where the integrity and accuracy of data are trusted. The process automates much of the manual verification work typically involved in the ELT lifecycle, allowing engineers to focus on more meaningful tasks and reducing the occurrence of post-deployment issues, thus enhancing overall productivity and confidence in data-driven decisions.

Data-driven organizations often face challenges with data quality, resulting in a lack of trust when data is incomplete or inaccurate, and recognizing that maintaining high-quality data is an ongoing journey rather than a destination is crucial. To enhance data quality, organizations should implement best practices such as automating data monitoring to proactively detect issues, utilizing tools like Slack for real-time alerts, and employing techniques like assertions, metric monitoring, and anomaly detection. Furthermore, improving the change management process is essential, including version-controlling code, understanding the impacts of data changes through data lineage and data diff tools, and establishing a thorough code review process to ensure changes are well-understood and bugs are minimized. By adopting these strategies, organizations can increase the reliability and value of their data, thus building trust with stakeholders and end-users over time.

Data diffing is a crucial process in data transformation that involves comparing two datasets to understand the impact of code changes on their shape and content. Depending on the format of the datasets, different approaches are recommended: using git diff for local files, dbt tests or Datafold Cloud's in-database data diffing for tables in a database, and Datafold Cloud's cross-database data diffing for tables across databases. A fictional scenario at a real estate listings company, Yillow, illustrates the application of data diffing, where a code adjustment to handle changes in address delimiters led to missing listings. This issue was identified using git diff, highlighting the importance of data diffing in debugging and ensuring data integrity. The text also discusses the limitations of git diff when dealing with unsorted data and suggests alternatives like using the diff command with additional options for more effective data comparison.

Apache Airflow is a widely adopted data pipeline orchestration framework, yet many teams overlook some of its most beneficial features for daily development, such as User-defined Macros, XCom, and Task Sensors. User-defined Macros allow for dynamic parameterization in data pipelines by enabling custom macros, which provide solutions for tasks like timezone adjustments and varying SQL clauses. XCom facilitates the passing of data between tasks by storing key-value pairs, although it can be complex to debug in intricate DAGs. Task Sensors, particularly the External Task Sensor, ensure task dependencies are maintained across DAGs, enhancing data integrity and allowing for coordination with external systems. Despite their power, these features need careful handling, and Airflow could benefit from expanding its dependency management capabilities beyond its current scope to improve integration with external services.

Data teams face the ongoing challenge of balancing velocity and quality, a problem long familiar to the software development community, as they enter the new year with limited budgets and tight deadlines. Emphasizing the importance of velocity, the text argues that the faster data products are delivered, the greater the impact on business decisions, especially under conditions of uncertainty and ambiguous requirements. However, data quality is equally critical, as trust in data is paramount; without it, users may abandon data-driven decision-making, diminishing the impact of the data team. The text highlights the need for data practitioners to prioritize both speed and reliability to maximize their contributions.