June 2023 Summaries

SodaGPT, now rebranded as SodaAI, is a pioneering generative AI tool designed to simplify data quality management by allowing users to express data quality checks without coding. Integrating the Soda Checks Language (SodaCL) with generative AI based on the Falcon-7b model, it translates natural language into production-ready checks, thereby reducing the workload on data engineers and empowering non-coders to manage data quality. This innovation supports the data mesh concept by treating data as a product and involving users in quality management, enhancing customer satisfaction and retention. SodaGPT prioritizes privacy and security by ensuring that data remains within the Soda platform, which is backed by SOC2 Type 2 certification. Currently available in preview, SodaGPT is set to evolve further, extending its capabilities to include more built-in and custom checks, promising continuous improvements in accuracy and integration with other Soda tools.

Group 1001, a technology-driven financial services company, faced challenges with data accuracy and consistency as their data volumes expanded. To address this, Gu Xie, the Head of Data Engineering, led a transformation to create a modern data stack that automated data operations, significantly boosting productivity. The adoption of Soda, a data quality tool, was pivotal, allowing the team to automate quality checks, reduce manual processes, and integrate seamlessly with existing technologies. This shift enabled the team to transition from reactive data management to proactive monitoring, enhancing data reliability and trust within the organization. By implementing a modular approach with tools like Fivetran, Coalesce, Dagster, and Soda, Group 1001 achieved enterprise-scale reliability without needing a large team. The transformation not only improved operational efficiency but also fostered a culture of data trust, where both analysts and engineers collaborate on defining and maintaining data quality standards. Looking forward, Group 1001 aims to deepen metadata-driven observability and democratize data usage across the organization.

Soda has introduced the Soda Library, an extension of Soda Core, to enhance data quality testing with new features and automation tools aimed at addressing users' challenges. The Soda Library includes capabilities like Check Suggestions, which offers a command-line interface assistant to help users create tailored data quality checks, and Group By and Group Evolution, which allows data teams to apply checks across groups dynamically. It also features Check Templates, enabling organizations to create reusable checks. While Soda Core remains free and its development will continue, Soda Library focuses on more advanced features linked to the company's commercial offerings. The transition to Soda Library is straightforward for users, ensuring no disruption in existing workflows. Soda is actively engaging with the community and planning further innovations, including AI-powered features, to support data-driven organizations.