October 2024 Summaries

Gretel has developed synthetic documents enriched with a wide variety of PII and PHI entities to improve entity detection without exposing real personal data. The gretelai/gretel-pii-masking-en-v1 dataset, created using Gretel Navigator, simulates real-world excerpts of documents filled with sensitive information across multiple industries and document types. By offering diverse scenarios, it pushes the boundaries of PII and PHI detection, giving developers confidence to fine-tune models while maintaining privacy compliance. The GLiNER models have been fine-tuned on this dataset, achieving significantly higher metrics compared to their base model counterparts. These models are ideal for applications in healthcare, finance, and more, ensuring accurate PII and PHI detection across diverse domains while complying with privacy regulations.

This blogpost discusses a method for fine-tuning large language models (LLMs) on specialized domains like healthcare while ensuring data privacy. The approach involves generating differentially-private synthetic text using Gretel's GPT model, which is then used to fine-tune LLMs for generating responses. Differential privacy provides formal guarantees that no training data will be ever extracted from the model, thus protecting sensitive information. The method was demonstrated by fine-tuning a Claude 3 Haiku model for generating clinical notes with the input of a transcript of conversation between a doctor and a patient.

The paper "GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models" by Mirzadeh et al. highlights important questions about LLMs' mathematical reasoning capabilities. It introduces GSM-Symbolic, an enhanced benchmark derived from the popular GSM8K dataset, and finds significant variability in model performance across different instantiations of the same question. The study also demonstrates that models are more sensitive to changes in numerical values than to changes in proper names within problems. However, its conclusions may not fully capture the complexity of the issue. Synthetic data generation techniques can address these challenges and push the boundaries of what AI models can achieve in mathematical reasoning tasks.