October 2022 Summaries

The article discusses the use of whylogs, an open-source data logging framework, for large scale data profiling. It explains how whylogs can be used to profile data with minimal overhead and highlights its mergability property which allows profiles of smaller pieces of a DataFrame to be added together to form a global dataset profile. The article also discusses the integration of whylogs with Fugue, an open-source project that brings Python, Pandas, or SQL code to Spark, Dask, or Ray. It explains how this integration allows users to maintain the same simple interface to generate profiles and scale data logging to big data frameworks like Spark. The article concludes by discussing various use cases of data profiling such as anomaly detection, drift detection, and data quality problems.

The whylogs library and WhyLabs platform provide a simple yet highly customizable solution for maintaining observability into image data. With native image profiling support re-introduced to whylogs v1.1, computer vision practitioners can now take full advantage of the simplified API, performance improvements, and visualization capabilities which debuted with whylogs 1.0. The platform allows users to extract the full value from these profiles with highly customizable anomaly detection, automatic notifications, and insightful visualizations.

WhyLabs has introduced a no-code integration option for its cloud storage data profiling tool, allowing users to bypass the need to integrate whylogs into their data pipeline. The first iteration of this solution is aimed at users who already have most of their data in cloud storage (starting with S3/AWS), don't want to invest any development time into profiling their data with whylogs, and don't mind permitting WhyLabs to ingest from their S3 bucket. The onboarding experience involves adding a policy to an S3 bucket that lets the company download data, adding a special tag to the S3 bucket, enabling S3 events directed at WhyLabs' SNS topic, specifying which column in your dataset represents time, and specifying the types of columns if needed.