May 2022 Summaries

whylogs is an open source tool for data logging that enables users to detect data drift, prevent ML model performance degradation, and validate the quality of their data. The v1 release brings a simpler API, new data constraints, new profile visualizations, faster performance, and a usability refresh. With whylogs, users can generate statistical summaries (termed whylogs profiles) from data as it flows through their data pipelines and into their machine learning models. These profiles enable users to track changes in their data over time, detecting data drift or data quality problems. The tool supports both tabular and complex data and runs natively in Python and JVM environments. It also supports batch processing (e.g., Apache Spark) and streaming (e.g., Apache Kafka). whylogs v1 is built for scale and optimized for massive data sets, with a more than 500x improvement in the speed of generating profiles for large datasets compared to the previous version.

Choosing the right data quality monitoring solution is crucial for maintaining high-quality data. An ideal monitoring solution should be self-service, dynamic, collaborative, holistic, automatable, privacy-preserving, and change-aware. Several open source and SaaS data quality monitoring tools are available in the market, including whylogs, Pandera, Great Expectations (GE), Deequ, WhyLabs, Metaplane, Monte Carlo, Soda Cloud, and Databand. When deciding whether to build or buy a data quality monitoring tool, consider factors such as business requirements, technical requirements, organization size, and implementation timeline.