January 2023 Summaries

Software testing has been integral to the software development life cycle for decades, but data quality testing has only recently begun experiencing similar advancements through a "shift-left" approach. This modern method involves addressing data quality assurance earlier in the data development process, paralleling software development practices, and aims to identify and resolve issues during the deployment and development stages rather than after production. The shift-left strategy in data engineering allows for proactive monitoring and automated testing, which helps prevent data quality issues that could lead to stakeholder distrust, incorrect decision-making, and reduced reliability of client-facing products. Treating data quality on par with software quality is essential because most data issues stem from bugs in the software that processes data. By implementing this approach, organizations aim to reduce the burden on data teams and the costs associated with maintaining data products.

Data testing is a critical component of ensuring data quality, particularly as companies increasingly rely on data-driven decision-making. It focuses on identifying and fixing data issues before they reach production, contrasting with data observability, which deals with data conditions post-production. Effective data testing is integral to maintaining data accuracy, completeness, consistency, and integrity, providing a safeguard against incorrect data entering production environments. The current state of data testing includes various methods such as the Table Scan, which involves visually inspecting data for anomalies; Quality Assurance, where stakeholders verify data accuracy, often focusing only on production data and metrics of interest; and mathematical approaches, which aggregate data to identify unexpected discrepancies. Although these methods are widely used, they each have limitations and can sometimes produce misleading results, emphasizing the need for robust, proactive testing strategies to ensure high data quality and reliability.