Introducing Soda Core: The New Way for Data Reliability

Over the past five years, the influx of software engineers into the data sector has highlighted the lack of established best practices in managing data products, leading to challenges in operationalizing these systems. The modern data stack has evolved significantly, driven by innovations like the data mesh, yet fundamental issues persist, such as the significant time spent by data scientists preparing data. Data engineers play a critical role in building and maintaining data pipelines, with a significant portion of their time dedicated to addressing data quality issues. A survey at the Snowflake Summit 2022 revealed that the primary bottleneck in resolving data issues is the lack of adequate tools and processes, which impacts business operations and trust in data. In response, Soda has developed an open-source framework, Soda Core, with a domain-specific language, SodaCL, to enhance data reliability and quality management. This framework aids data engineers in automating the detection and resolution of data issues, providing accessible and human-readable tools that engage a wide range of team members in maintaining data quality. With the community's positive feedback and contributions, Soda Core aims to simplify data engineering tasks and improve data reliability across the board.