July 2023 Summaries

An end-to-end data cleaning workflow is crucial for preparing tabular data for AI and ML projects, as emphasized by the principle "garbage in, garbage out." The text outlines various steps in the data cleaning process, using a modified Adult Census Income dataset to demonstrate common issues such as standardizing empty values, removing duplicate records, handling missing data, and addressing field and record level outliers. Techniques include using machine learning-based imputation methods like MissForest to fill in missing data and employing IsolationForest for detecting outliers. Redundant fields, such as those that are highly correlated or contain constant values, are removed to ensure a more efficient and accurate model training process. The article highlights the importance of these steps in enhancing the quality of synthetic data generated by Gretel’s models, ultimately leading to more successful AI/ML outcomes.

A major hospital system used Gretel, an AI platform, to generate synthetic medical data for forecasting staffing and resource needs in their emergency care unit and identifying emerging trends in outbreaks. Synthetic data offers several advantages over traditional approaches to maintaining data privacy, including provable anonymity, portability and shareability, and flexibility for exploratory data analysis. The hospital used Gretel's synthetic data models to create artificial versions of their training data with various levels of privacy protections. Comparing the performance of a model trained on synthetic data versus real-world data showed that synthetic data created by Gretel ACTGAN outperformed real world data in predicting the length of a patient's emergency room stay based on their initial diagnosis.

The text describes a process of anonymizing 7 terabytes of data using Gretel Transform in a hybrid cloud environment. The main challenge was to ensure that the data and models remain within the customer's cloud throughout the process, as some customers work in heavily regulated industries with stringent data residency requirements. To address this, Gretel developed a Kubernetes deployment strategy for its hybrid workflows. This approach allowed them to scale their hybrid workflows based on the number of nodes they could run in a customer's environment and keep all data within the customer's cloud during the process. The text also provides details about preparing the data, running training and transform jobs, and loading the transformed data into a lower environment for testing purposes.