December 2022 Summaries

Soda Cloud has introduced a new feature called Soda Incidents, aimed at enhancing data operations by efficiently managing data reliability and quality issues. This feature addresses the challenges of data downtime, which can lead to business disruptions, productivity loss, and increased skepticism in data trust. Soda Incidents streamlines the process of detecting, triaging, diagnosing, and resolving data issues throughout the data product lifecycle, emphasizing teamwork, defined roles, and clear communication. The platform integrates with tools like Slack for real-time collaboration, allowing teams to respond swiftly to data quality alerts and manage incidents effectively. By incorporating best practices from incident management, Soda Incidents enables organizations to configure workflows to meet specific business requirements, ensuring data uptime and continuous improvement through retrospectives. The integration with existing issue trackers like Jira and OpsGenie, along with upcoming features such as leveraging dbt test results, further enhances Soda Cloud's utility for data teams.

Soda Spark has been deprecated and replaced by the Soda Library, which connects Soda to Apache Spark, offering a modern approach to data testing, monitoring, and reliability for engineering teams using PySpark DataFrames. This tool is designed to help data and analytics engineers maintain high data quality in data-intensive environments by providing an API for extracting data metrics and column profiles via YAML configuration files. When connected to a Soda Cloud account, Soda Spark enables teams to configure alerts for failed tests, facilitating quick issue resolution. The tool simplifies the process of writing declarative data quality tests for Spark DataFrames and is compatible with various data workloads, engines, and environments, such as Kafka, AWS S3, and Google BigQuery. With easy installation via pip and integration into platforms like Databricks, Soda Spark supports collaborative efforts through Soda Cloud, which offers features like anomaly detection, schema evolution monitoring, and visualization of test results. Soda's open-source and SaaS offerings emphasize data reliability, with a strong community backing, including major contributors like Disney and HelloFresh.