August 2023 Summaries

The text discusses best practices for using GitHub features such as branches, branch naming conventions, commit messages, and pull requests to maintain the integrity of data models in analytics workflows. It emphasizes the importance of using branches to separate code changes, which prevents disruptions to the main code base and allows for easier identification and rollback of updates. Establishing branch naming conventions, particularly by associating them with Jira ticket numbers, helps maintain consistency and accountability within a team. Descriptive commit messages are crucial for tracking code changes and facilitating potential reversions, with examples illustrating the difference between vague and informative messages. The text also outlines the procedure for creating pull requests, highlighting their role in ensuring code quality by requiring peer review before merging changes into the main branch, thus reducing the risk of introducing faulty code into production.

Datafold has launched a new Looker integration for Datafold Cloud, designed to preemptively catch data quality issues before they impact business intelligence (BI) tools crucial for decision-making. This integration provides data teams with enhanced visibility into Looker assets such as Explores, Views, Dashboards, and Looks that could be affected by changes in dbt code. Through Datafold Cloud's comprehensive column-level lineage and PR impact analysis comments, teams can identify potential data problems before they reach stakeholders. The integration addresses a common challenge where BI tools and dbt transformations lack seamless integration, often resulting in reactive fixes and frustration. By bridging this gap, Datafold aims to ensure that dbt code changes do not introduce inaccurate data into dashboards, thereby facilitating proactive data management and improving the workflow between dbt development and BI reporting.

Looker and Tableau are prominent business intelligence (BI) tools, each with unique strengths and weaknesses, leaving businesses to choose based on their specific needs. Looker, part of Google Cloud Platform, offers a code-based approach with LookML, enabling strong data governance and accessible data analysis through its Explores, Views, Looks, and Dashboards. It facilitates self-service analytics but comes with a learning curve, especially for users unfamiliar with its dimensions and metrics. Tableau, owned by Salesforce, excels in offering complex, highly customizable data visualizations, making it popular in enterprise markets. It requires stronger technical skills for development but is intuitive for business users to interact with. Both tools face challenges in data governance at scale, often struggling with tracking data usage across numerous reports. Datafold Cloud's column-level lineage integration aims to bridge the gap between data transformation and BI tools, providing insights into data usage and potential impacts of upstream changes. Ultimately, the choice between Looker and Tableau depends on the technical level of end users, primary use cases, and the comfort level with users writing their own queries.

Datafold addresses prevalent challenges in data quality testing faced by data teams, highlighting the limitations of traditional methods such as assertion-based tests in identifying unforeseen changes in data. The text recounts personal experiences of data quality mishaps, illustrating how even minor code changes can lead to significant inaccuracies and distrust among business users. It identifies five common pain points, including the slowing down of dbt development and the erosion of trust between data teams and business users due to inaccurate data. Datafold introduces data diffing as a solution, a method that compares tables to detect changes before they impact production, thereby enhancing data quality testing and allowing teams to work with greater confidence and efficiency. This proactive approach aims to restore trust and streamline development processes by ensuring data integrity and transparency throughout the data pipeline.

The text explores the implementation and benefits of Continuous Integration (CI) in dbt projects, emphasizing its importance in modern data teams. CI is a software engineering principle that helps ensure new code is ready for production by integrating changes, running automatic tests, and maintaining code standards. The process involves multiple environments—production, development, and staging—to keep untested data separate from production data, thus reducing the risk of errors. The CI pipeline typically includes steps like running dbt compile, building projects in a staging environment, conducting dbt tests, using SQL linters, and performing data quality checks. Implementing CI can accelerate team velocity, improve collaboration, ensure stakeholder trust, and enhance the overall quality of life for analytics engineers by minimizing error-prone deployments and reducing burnout. The text also provides guidance for setting up CI in dbt Cloud and dbt Core, highlighting the positive impact on data-driven decision-making and team productivity.

Datafold's data diffing is a process that compares two tables to identify changes, similarities, additions, or removals of values, akin to a git code diff but for data tables. This technique is particularly useful for validating parity between development and production environments or across different data warehouses, offering both high-level overviews and detailed value-level insights. Data diffing helps users swiftly detect structural differences such as schema and primary key variations, as well as row-level discrepancies, enabling them to understand the impact of code changes on data. Although traditional testing frameworks like dbt catch expected changes, they may overlook unexpected modifications; therefore, data diffing complements existing tests by highlighting unforeseen differences that manual checks might miss. This practice is increasingly adopted by teams engaged in data migrations or replication to ensure data accuracy and integrity, providing a clear and efficient way to manage data quality amidst growing data and project complexity.

Datafold offers two solutions for data diffing: an open-source Python package called data-diff and a SaaS platform known as Datafold Cloud. Data diffing, akin to git code diffing but for database tables, involves comparing two tables to identify changes in value, schema, or row count. The open-source data-diff is ideal for individual developers or small teams needing ad hoc comparisons, particularly during development or data migrations, and integrates with dbt for enhanced model comparison. In contrast, Datafold Cloud is tailored for larger teams requiring automated data diffs, comprehensive column-level lineage for dependency analysis, and secure, compliant operation in environments like SOC 2, HIPAA, and GDPR. It supports a variety of workflows, including CI-integrated diffing, offering detailed insights into how changes impact data ecosystems, including BI tools and data apps. The best choice between the two depends on team size, data complexity, need for automation, and compliance requirements.

In the realm of engineering and data infrastructure, the persistent dilemma of choosing between building in-house solutions and purchasing off-the-shelf products is highlighted, with the latter often being the more cost-effective and efficient choice. The text emphasizes that attempting to replicate vendor solutions, which have been honed over years by specialized teams, can lead to significant resource allocation problems, including labor costs and opportunity costs of time. While engineers may feel the urge to build custom solutions due to the complexity and appeal of such projects, the text advocates for focusing engineering efforts on higher-value tasks like developing user-facing features and machine learning automation. The narrative warns against the potential pitfalls of selecting poor vendors, despite the general recommendation to buy rather than build, and suggests that a well-chosen product can ultimately save time and resources while delivering effective results.