May 2022 Summaries

Using an open-source implementation of GPT-3, this text discusses the process of augmenting machine learning datasets with synthetically generated text and labels, offering a scalable, fast, and cost-effective alternative to traditional data augmentation methods. The approach involves fine-tuning a GPT model on a financial intent classification dataset named `banking77`, which contains 13,083 customer service queries labeled with 77 intents. By encoding intent labels and text into a single field and using conditional generation, the model generates new annotated examples for the intent classes. The text highlights the advantages of synthetic data, such as improved privacy and reduced costs compared to real-world data collection and annotation, and cites a Gartner prediction that synthetic data will dominate AI data use by 2030. The process employs GPT-Neo via Gretel.ai’s APIs, but other GPT models from the HuggingFace repository can also be used. The synthetic data is generated by seeding the model with class examples, and the output is formatted into a tabular format for analysis. The text concludes by expressing enthusiasm for the capabilities of generative pre-trained transformers and invites readers to explore further resources and participate in the community.

Gretel explores state-of-the-art image synthetics for business documents using diffusion models, a potential replacement for Generative Adversarial Networks (GANs). Diffusion Models have shown to generate high-quality images with great diversity and perform well on various data types. The company recently explored how well Diffusion Models perform at document synthesis, particularly in generating realistic-looking receipts from a small dataset of 300 images. They found that the model not only captured shapes, lighting, and variations but also wrote coherent text. This technology can be used for important business applications such as creating synthetic images of documents for analysis or training downstream machine learning systems.

Model Soup is an ensembling technique that improves overall performance by averaging the weights of multiple models instead of combining their individual outputs. This technique has been found to perform better than any individual model on benchmark datasets like ImageNet. Gretel, a company working with synthetic data generation, explored this method to improve model performance on smaller datasets and found promising results. However, they also observed that souping more than five models or certain types of models led to poor performance. Further exploration is needed to determine the effectiveness of Model Soup in different scenarios and identify patterns in its performance.

The blog post introduces two new notebooks from Gretel—multi-table transform and multi-table synthetics—that offer streamlined processes for anonymizing data in relational databases while preserving the referential integrity of primary and foreign keys. These notebooks can be used independently or in tandem, depending on the level of privacy required. The transform notebook focuses on de-identifying sensitive information, making it suitable for pre-processing or demo environments, while the synthetics notebook augments data and is ideal for statistical or machine learning analyses. Both notebooks operate on a mock e-commerce database and offer configurations for transforming and synthesizing data, ensuring the relationships between tables are maintained. The process includes training models, generating data, and ultimately loading the final anonymized datasets back into the database, with the synthetics notebook providing additional performance reports. This approach allows users to anonymize and augment data effectively, offering robust privacy protections while ensuring the statistical integrity of the dataset.