October 2021 Summaries

In her guest blog, Sarah Krasnik, Lead Data Engineer at Perpay, emphasizes the importance of thorough quality assurance (QA) in analytics engineering to prevent production issues and maintain data integrity. She highlights the need for a programmatic approach to data QA, advocating for the use of "Data Diff" to compare datasets across environments, such as staging and production, to identify discrepancies before deployment. Krasnik discusses two key audit types: assessing the total number of rows and primary key changes, and analyzing value distributions to ensure changes align with business expectations. By conducting these audits, teams can detect potential issues early, communicate changes effectively, and avoid unexpected surprises in dashboards, thus enhancing the reliability of analytics outputs.

As privacy regulations like GDPR become increasingly relevant, organizations must ensure that Personally Identifiable Information (PII) is properly managed within their data pipelines to maintain compliance and protect consumer data. PII, which includes information like names, email addresses, and social security numbers, can inadvertently end up in places it shouldn't, such as aggregate analytics or chatbot training models. Tools like Datafold facilitate the tracking and management of PII by offering column-level lineage, allowing data engineers to trace data through upstream and downstream processes and ensure sensitive information is not improperly retained. Datafold's interface supports tagging and tracking of PII, thereby enabling teams to collaborate effectively in maintaining data privacy and security. This process is not a one-time task but part of ongoing efforts to comply with evolving privacy requirements, and Datafold's integration with tools like dbt and Snowflake aids in the seamless implementation of these best practices.