How we automated data validation

Amanda Fioritto details the Hex data team's journey to automating data validation processes, culminating in the development of an AI-powered audit-helper skill that significantly enhances the efficiency and accuracy of their validation workflows. The text describes how the team initially used dbt's audit_helper package for standardized testing but later sought more flexibility and visibility, leading them to build the Hex Audit Helper app. As AI tools advanced, they developed an agent skill to conduct end-to-end audits with principles of standardization, flexibility, and traceability, ensuring that validation remains a human-guided process despite automation. The audit-helper skill incorporates explicit guidelines to handle large models, reduce noise, and avoid assumptions by requiring evidence for changes, thereby making validation quicker and more comprehensive. The text emphasizes the role of AI as a supportive partner rather than a replacement for human judgment, aiming to make data validation more seamless and impactful, ultimately enabling the data team to focus on higher-value tasks and improve decision-making across the organization.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.