October 2021 Summaries

Organizations are increasingly focusing on creating and operationalizing data products to enhance business outcomes, leading to a need for robust data quality monitoring to prevent silent data issues that can negatively impact these products. Soda's Schema Evolution Monitor, part of an automated monitoring suite, addresses this by detecting and alerting on changes in dataset schemas, such as additions, deletions, or type changes of columns, without requiring configuration. This ensures that data teams are promptly notified of potential issues, allowing them to mitigate downstream impacts like incorrect reports or flawed machine learning models. While the current system effectively monitors schema changes, Soda is developing a new language to enhance data reliability by enabling schema testing at various stages of data processing, which will allow for quarantine of problematic data. This upcoming feature is expected to significantly improve the ability to maintain data quality and trust in data products.

Organizations are undergoing a transformative shift in their approach to data, necessitating a reevaluation of team structures to optimize data management and value creation. This evolution is explored through a series in partnership with Soda, using football as an analogy to illustrate the composition and function of a 'Data Dream Team'. The series identifies three critical teams: the operations team, akin to a football defense, ensures smooth software production and data reliability; the data engineering team, like the midfield, is responsible for creating data products and pipelines; and the data science team, compared to forwards, leverages advanced analytics to generate business value. Leadership, represented by a football manager, plays a crucial role in aligning these teams under a unified strategy, with a single executive often leading to better cohesion. An effective data team requires a balance of roles and skills, and organizations often falter by lacking one of these essential components. The series also includes podcasts featuring industry experts discussing strategies for building and nurturing successful data teams.