June 2021 Summaries

Soda's Time Series Anomaly Detection is an automated tool designed to enhance data quality and trust within organizations by identifying unusual data points in metrics that unfold over time, such as sales figures or valid value percentages. This feature, part of the Soda Data Observability Platform, applies machine learning algorithms to understand and predict data patterns, flagging anomalies without requiring complex configurations or threshold settings. It empowers data teams to address data quality issues proactively, facilitating root cause analysis and reducing the operational risks and costs associated with poor data quality. By learning from user feedback, the system adapts to specific business contexts, minimizing false alerts and enhancing focus on significant anomalies. Soda's focus on collaboration ensures that alerts are directed to the appropriate team members for timely investigation, supporting various industries like e-commerce, manufacturing, and finance in optimizing their data-driven decision-making processes. The platform aims to drive automated processes and insights by developing features that suggest monitors, group alerts, and analyze diagnostic data for root cause identification, offering a practical solution for organizations seeking to maintain high data quality standards.

HelloFresh recently sponsored a global hackathon, supported by Soda, aimed at creating a "Standard Data Quality Dashboard" to enhance data literacy and trust within the organization. The event brought together cross-market teams of employees who were tasked with designing dashboards that would simplify data quality reporting and enable easy comparison of data assets, thus reducing the time and resources spent on reacting to data quality issues. The teams focused on six key dimensions of data quality—accuracy, completeness, consistency, timeliness, uniqueness, and validity—and incorporated features that allowed users to gain both high-level overviews and deep insights into datasets. The winning team, "Tricolour Kiwis," proposed a dashboard that weights the importance of each dimension specific to each data asset, demonstrating the importance of tailored data quality metrics. Soda's Data Observability Platform played a crucial role in underpinning the data quality metrics, highlighting the significance of accessible data monitoring in fostering transparency and informed decision-making across organizations. The hackathon emphasized the value of making data monitoring accessible and encouraged other organizations to explore similar initiatives using Soda's resources and platforms.

Soda Cloud is a platform designed to address and manage silent data issues by offering data teams a comprehensive system for data quality, combining predictive capabilities with a user-friendly, rules-based approach. These silent data issues, which often go unnoticed until datasets are used in decision-making processes, can have significant downstream impacts. The platform provides end-to-end observability, enabling teams to discover, prioritize, and resolve data issues collaboratively and earlier in the data lifecycle. It supports various team members, from data platform engineers to product managers, by allowing them to set up complex data validations without extensive coding knowledge. Soda Cloud emphasizes a team-based approach to data quality, integrating tools and workflows to suit the needs of different roles, fostering a culture of data ownership and collaboration. It addresses the scalability issues of traditional data testing setups by offering a modern, centralized platform that facilitates real-time collaboration, monitoring, and resolution of data issues, ultimately aiming to build trust in data across organizations.

Silent data quality issues pose a significant challenge for data teams, often going undetected and leading to downstream impacts, which necessitates the adoption of effective data management strategies. Transitioning from software engineering, the founders of Soda identified the need for systematic data testing and monitoring, prompting the creation of Soda SQL, an open-source tool designed to integrate with existing data workflows to define and ensure data quality by running SQL-based tests. Soda SQL, which has evolved into Soda Library, allows teams to detect invalid, missing, or unexpected data and stop data pipelines if issues are identified. Complementing Soda SQL, Soda Cloud is a web application that enhances visibility and collaboration among data teams by providing a platform for monitoring data metrics over time, enabling anomaly detection and empowering non-technical users to engage with data quality through a user-friendly interface. Through integrations with communication tools like email and Slack, Soda Cloud ensures timely alerts for diagnosing and resolving data issues, promoting a collaborative approach to maintaining data integrity and making data quality a shared responsibility across organizations.

Data observability platform Soda aims to tackle the issue of silent data quality problems that often go undetected, posing significant risks to data-driven products. Having transitioned from software to data engineering, the founders initiated this endeavor to implement strategies akin to software testing and monitoring within the data domain. Soda introduced Soda SQL, an open-source tool launched in February 2021, which allows data engineers to define what constitutes good data quality, facilitating testing and monitoring within existing data workflows. This tool, which utilizes YAML config files and SQL for data testing, helps detect and address invalid, missing, or unexpected data, potentially halting pipelines to prevent further issues. Additionally, Soda Cloud, a web application, extends Soda SQL's capabilities by offering a collaborative platform where metrics and test results are visualized over time, enabling even non-technical users to participate in monitoring and ensuring data quality. This approach fosters a collaborative environment where data teams can preemptively address data issues, aligning stakeholders on quality expectations through integrations with communication tools like email and Slack.