Scaling Data Quality Across a Modern Data Stack

Group 1001, a technology-driven financial services company, faced challenges with data accuracy and consistency as their data volumes expanded. To address this, Gu Xie, the Head of Data Engineering, led a transformation to create a modern data stack that automated data operations, significantly boosting productivity. The adoption of Soda, a data quality tool, was pivotal, allowing the team to automate quality checks, reduce manual processes, and integrate seamlessly with existing technologies. This shift enabled the team to transition from reactive data management to proactive monitoring, enhancing data reliability and trust within the organization. By implementing a modular approach with tools like Fivetran, Coalesce, Dagster, and Soda, Group 1001 achieved enterprise-scale reliability without needing a large team. The transformation not only improved operational efficiency but also fostered a culture of data trust, where both analysts and engineers collaborate on defining and maintaining data quality standards. Looking forward, Group 1001 aims to deepen metadata-driven observability and democratize data usage across the organization.