June 2021 Summaries

Data quality testing is a crucial component in data engineering workflows, ensuring the accuracy and reliability of data used for business decisions, similar to how software engineers test applications. By integrating automated data testing within continuous integration and continuous delivery (CI/CD) frameworks, data engineering teams can maintain data integrity, reduce manual errors, and respond swiftly to data quality drifts. Tools like GitHub and CircleCI facilitate version control and automate testing processes, allowing for staging and production environment comparisons to be seamlessly conducted. Automated data testing, through tools like Great Expectations and dbt, enables data teams to enforce data quality gates, ensuring only validated data updates business intelligence dashboards. This automation not only enhances the efficiency of data workflows but also builds organizational trust in data-driven decision-making by preventing the propagation of inaccurate data.

The fourth Data Quality Meetup, held on May 20th, 2021, featured seven guest speakers and over 120 live participants, focusing on strategies for improving data quality and management across various industries. Key discussions included the importance of investing in robust change management processes to prevent data failures, as outlined by Gleb Mezhanskiy, and the implementation of Verity by Lyft for monitoring data quality at scale. Shopify's Michelle Ark highlighted the use of dbt for efficient data pipeline development, while Caitlin Colgrove from Hex emphasized the need for data security measures against SQL injections. Alex Viana from HealthJoy introduced a "working backwards" approach to data projects to manage stakeholder expectations and reduce costs. Additionally, a panel discussion explored themes such as reliable change management, the balance between data empowerment and control, and maintaining development speed in regulated industries. These insights underscore the critical importance of proactive data management, security, and strategic planning in contemporary data environments.