September 2021 Summaries

Data lineage tools like Amundsen and Data Hub are essential for visualizing data transformations within pipelines, offering table-level insights for analysts and scientists. However, when deeper insights are needed, such as the origin of data in BI reports or the impact of altering a column, column-level lineage becomes crucial. Column-level lineage provides a detailed view of data flow, helping track changes or usage of sensitive data without extensive SQL code reviews. Datafold offers an intuitive interface for this purpose and integrates with existing tools in the modern data stack through its GraphQL API, facilitating the incorporation of lineage data into other catalogs. The blog highlights a practical example of integrating column-level lineage data into Amundsen using a beer-themed data pipeline, modeled in dbt and leveraging BigQuery, allowing users to gain insights into brewery locations and beer styles efficiently. While dbt natively supports table-level lineage, this example demonstrates how to scale lineage tracking using Datafold's API and Amundsen's Databuilders, ensuring seamless metadata integration and enhanced data visibility.

Data quality management is crucial for ensuring accuracy and usability of data, especially at scale, as highlighted by the challenges faced by companies like Lyft, Shopify, and Thumbtack. Lyft addressed data quality issues by creating Verity, a proprietary tool that conducts data quality checks and can block data consumption if errors are detected, achieving significant coverage for their datasets. Shopify faced scalability issues with their dbt modeling tool and introduced Seamster, an in-house framework for SQL unit testing, allowing for quick detection and resolution of code flaws. Thumbtack automated their data quality assurance process by integrating Datafold's Data Diff tool, which streamlined their data change verification process and improved productivity by automating regression checks. These examples illustrate that data quality management is an ongoing process requiring continuous improvement and a strong data culture, facilitating better data governance and observability.

The fifth Data Quality Meetup featured a presentation by Ahmed Elsamadisi, CEO of Narrator AI, on the concept of creating a single source of truth through an Activity Schema, which transforms raw data into a comprehensive event-based table to ensure consistent data representation. The schema, accessible via open-source, enables users to map relevant data into an activity stream using simple SQL snippets and facilitates querying through 11 time-based operators, which are complex to code but simplified by an app that auto-generates SQL queries. Another presentation by Katya Ogai, Director of Analytics at AppFolio, explored different organizational structures for analytics teams—centralized, decentralized, and hybrid—each with its advantages and challenges, highlighting the hybrid model as her preferred choice for balancing technical and business mentorship while promoting cross-collaboration and synergies within the organization.

After joining Datafold, the author was tasked by CEO Gleb with pursuing SOC 2 compliance to demonstrate the company's commitment to data security and integrity. SOC 2, created by the AICPA, is a security audit that evaluates an organization's data protection controls. Datafold achieved SOC 2 Type 1 compliance, which assesses security practices at a specific time, and aims to achieve Type 2 compliance, requiring a longer evaluation period. The process involved revising policies, using tools like Vanta for guidance, and working with auditors to ensure controls were effective and auditable. The author learned the importance of aligning policies with business risk rather than making them overly strict. During the audit, clear documentation and cooperation with auditors were crucial. Datafold plans to pursue further compliance initiatives, continue improving security policies, and invest in training as the company grows.