October 2024 Summaries

The blog post discusses the advantages of fine-tuning small open-source models using synthetic data over GPT-4o, particularly in scenarios with limited training data. Predibase's synthetic data generation workflow allows for effective training of models like Llama-3.1-8b with minimal real data, significantly reducing costs compared to GPT-4o. Various synthetic data generation methods are explored, including K-shot prompting, single seed example, single pass, and the mixture of agents (MoA) approach, each offering different benefits in terms of context, specificity, and dataset distribution. The MoA method, while more complex and costlier upfront, excels in producing high-quality datasets by balancing context and specificity, making it advantageous for fine-tuning small language models. Experiments show that fine-tuned models on synthetic data can surpass GPT-4o's performance, especially as the dataset size increases, and alternative tools like Gretel.ai also provide effective synthetic data solutions. The post concludes with a tutorial on using Predibase for synthetic data generation and fine-tuning, underscoring the importance of high-quality seed data and the benefits of fine-tuning over K-shot prompting.

Predibase has unveiled its Inference Engine, an advanced platform designed to optimize the deployment of fine-tuned small language models (SLMs) for enterprises, addressing challenges such as cost-efficiency, scalability, and performance in AI production environments. Utilizing innovations like Turbo LoRA and LoRA eXchange (LoRAX), the Inference Engine enhances throughput, reduces infrastructure costs, and supports the deployment of numerous SLMs on a single GPU, thereby minimizing resource requirements. The platform's capabilities include FP8 quantization for memory efficiency, GPU autoscaling to adjust resources in real-time based on demand, and multi-region high availability to ensure uninterrupted service. These features collectively aim to streamline AI operations by offering enterprises a flexible, secure, and cost-effective solution for serving fine-tuned SLMs, whether through Predibase’s managed cloud or within their own virtual private cloud infrastructure.

Checkr, a technology company founded in 2014, specializes in modernizing background checks using AI and machine learning to improve efficiency, inclusivity, and transparency. At the LLMOps Summit in San Francisco, a Checkr representative shared their experience in building a language model-based system to automate the adjudication of background checks, significantly reducing costs and manual reviews. The team experimented with various large language models (LLMs), including GPT-4 and fine-tuned smaller models like Llama-2-7b, achieving improved accuracy and efficiency with reduced latency and cost. They utilized Predibase for fine-tuning, which provided the best performance and reliability, enabling Checkr to scale and expand its use cases while maintaining compliance and transparency in hiring practices. By leveraging Low-Rank Adaptation (LoRA) and Predibase's infrastructure, Checkr successfully optimized their models for production, resulting in a 5X cost reduction compared to previous methods.