November 2023 Summaries

Gretel provides a way to generate synthetic data using its hybrid solution, keeping sensitive data within the boundaries of customers' own cloud tenants. The core compute service powering Gretel Hybrid is Kubernetes, which supports a "bring your own compute" approach across major cloud providers like AWS, Azure, and GCP. Terraform is used to enable rapid hybrid deployments and feature delivery. In this post, the author walks users through the steps of deploying Gretel Hybrid into their own AWS environment and generating synthetic health data using a demo dataset. The guide covers prerequisites, deployment, generating synthetic data, cleanup, and further resources for exploration.

During the Gretel Demo Day, advancements in synthetic data generation were shared with the AI community. Key highlights included the introduction of Model Playground and Tabular LLM, a model designed for generating tabular data. The event also showcased no-code solutions for automating synthetic data tasks within existing data pipelines. Synthetic data is gaining momentum as it enables secure analytics, model training, and privacy compliance. Policymakers are leveraging synthetic data to enhance regulatory oversight, with the UK's FCA using it in its digital sandbox program. The U.S. President's Executive Order on AI also emphasizes the adoption of privacy-enhancing technologies like synthetic data.

AWS and Gretel have launched a strategic collaboration to support privacy-centric generative AI development through the Synthetic Data Accelerator Program. The program aims to provide comprehensive solutions for training, testing, and fine-tuning machine learning models without compromising individual privacy or trade secrets. Participants will gain access to direct support from technical experts, early access to Gretel's Tabular LLM, and opportunities to share research and insights at a generative AI workshop series. The program is open to startups and enterprises across various industries such as financial services, healthcare, and the public sector. Synthetic data offers numerous benefits including mitigating privacy risks, augmenting limited data supplies, simulating edge cases, and complying with regulations like GDPR, CCPA, and HIPAA.

AWS and Gretel have launched a strategic collaboration to support privacy-centric generative AI development through the Synthetic Data Accelerator Program. The program offers direct support from technical experts, early access to Gretel's Tabular LLM, and opportunities to share research at a generative AI workshop series. Synthetic data is in high demand due to its ability to maintain statistical insights while mitigating privacy risks, augmenting limited data supplies, simulating edge cases, and complying with regulations like GDPR, CCPA, and HIPAA. The collaboration aims to foster an ecosystem of responsible AI innovation by enabling teams to safely test, train, and fine-tune proprietary large language models (LLMs) and other AI applications using synthetic data.

This blog discusses how Gretel's Text SQS can be used to optimize the Llama-2 model during fine-tuning and generation of synthetic text data. The experiment uses a sample of 1000 records from the SAMSum dataset, which is a text summarization dataset containing dialogues and human-written summaries in English. Gretel's Text Synthetic Quality Score (SQS) is used to evaluate the quality of generated text records during fine-tuning. Results show that increasing steps in lower ranges improves learning and results in better text SQS, with scores above 80 considered "excellent." The experiment demonstrates how Gretel's synthetic text evaluation score can precisely measure the quality of generated records, allowing developers to focus on building and uncovering insights.