Case Study: CollegeVine Detects Incorrect Claims from AI Agent Responses at scale with Refuel

CollegeVine, a platform enabling higher education institutions to deploy AI agents for operational interactions, sought to improve the accuracy and trustworthiness of AI-generated claims due to issues with hallucinations and outdated training data. Initially using GPT-4o-mini, which achieved sub-85% accuracy, CollegeVine faced challenges with latency and accuracy using larger models like GPT-4o. Partnering with Refuel, CollegeVine used human-verified, labeled data to fine-tune the Refuel-LLM model, resulting in 93% accuracy and significant reductions in errors, speed, and costs. The collaboration allowed CollegeVine to seamlessly integrate the model into their production pipeline, achieving 50% fewer errors and 40% faster speeds, with the solution serving over 2 billion tokens per day and requiring minimal engineering effort. Chris Coffey, CTO at CollegeVine, expressed enthusiasm for deploying fine-tuned models that surpass GPT for classification tasks, highlighting the efficiency and scalability achieved through their partnership with Refuel.