January 2025 Summaries

CollegeVine, a platform enabling higher education institutions to deploy AI agents for operational interactions, sought to improve the accuracy and trustworthiness of AI-generated claims due to issues with hallucinations and outdated training data. Initially using GPT-4o-mini, which achieved sub-85% accuracy, CollegeVine faced challenges with latency and accuracy using larger models like GPT-4o. Partnering with Refuel, CollegeVine used human-verified, labeled data to fine-tune the Refuel-LLM model, resulting in 93% accuracy and significant reductions in errors, speed, and costs. The collaboration allowed CollegeVine to seamlessly integrate the model into their production pipeline, achieving 50% fewer errors and 40% faster speeds, with the solution serving over 2 billion tokens per day and requiring minimal engineering effort. Chris Coffey, CTO at CollegeVine, expressed enthusiasm for deploying fine-tuned models that surpass GPT for classification tasks, highlighting the efficiency and scalability achieved through their partnership with Refuel.

RefuelLLM-2-mini, a new 1.5 billion parameter model, joins the Refuel-LLM family, demonstrating superior performance in data labeling tasks when compared to other models like Phi-3.5-mini and Qwen2.5-3B. Designed for data labeling, enrichment, and cleaning, RefuelLLM-2-mini is built on the Qwen2-1.5B base model and trained on a diverse corpus of over 2,750 datasets, including both human-annotated and synthetic data. It achieves high output quality and well-calibrated confidence scores, with low latency performance. The model is accessible through Refuel Cloud and is open-sourced on Hugging Face under a CC BY-NC 4.0 license, thanks to contributions from various open-source initiatives and infrastructure support from organizations like Mosaic and GCP.