October 2023 Summaries

dbt Cloud enhances the capabilities of the Data Build Tool (dbt) by offering features that address common limitations found in local dbt model execution and orchestration tools. It provides robust notification systems for data pipeline monitoring through platforms like Slack or email, allowing users to customize alerts based on environment and run types to avoid unnecessary clutter. dbt Cloud also simplifies the hosting and sharing of documentation by automatically generating and updating project docs, making them accessible to all authorized users, including non-technical business users. The platform facilitates source freshness checks, ensuring critical data models can be refreshed even if some sources fail. A notable feature is the integration with continuous integration (CI) systems, allowing users to test pull requests before merging to maintain data quality, using a streamlined approach that only runs modified models. This combination of features makes dbt Cloud a powerful tool for managing and maintaining reliable data workflows.

Software engineers have successfully automated their workflows, which has greatly increased productivity and is influencing other fields, including data analytics. Data professionals are adopting similar practices, particularly automated regression testing, to ensure data integrity and prevent disruptions caused by changes. This approach helps detect unexpected issues that could affect downstream data applications, as illustrated by an incident experienced by Datafold's CEO at Lyft. Regression testing differs from traditional assertion-based tests by uncovering unknown or unexpected issues in datasets. Tools like Datafold, when integrated into continuous integration (CI) pipelines, allow for precise data comparison and detailed analysis, offering confidence in data changes before they are deployed to production. By incorporating CI and automated testing, data teams can standardize their workflows, documenting and understanding the impact of every code change, while maintaining involvement through code reviews. This method is recommended for managing dbt projects, as it can either flag potential issues or halt code changes to ensure high-quality data management.

Gartner's Data Quality Market Survey highlights the significant financial impact of poor data quality, costing businesses an average of $15 million annually, with nearly 60% failing to measure these costs. The cases of Samsung and Uber serve as cautionary tales, illustrating the importance of implementing robust data quality assurance processes. Samsung experienced a costly "fat-finger" error in 2018, where an employee mistakenly issued shares instead of currency, resulting in a $300 million loss. Similarly, Uber's accounting error, due to an oversight in commission calculations, led to underpaid driver compensation, costing the company millions. Both instances underline the necessity for companies to prioritize high-quality data and establish preventive measures, such as systematic data verification and quality management practices, to mitigate potential risks and financial losses. Harvard Business Review emphasizes that only a small percentage of companies meet basic data quality standards, advocating for organizations to integrate data quality into their key performance indicators and foster a culture where data integrity is a shared responsibility across all levels.