Building production AI for regulated industries with a leading digital insurer

A leading digital insurer partnered with Baseten to develop a state-of-the-art AI system that efficiently handles customer insurance queries while maintaining strict compliance with Financial Conduct Authority (FCA) regulations and delivering sub-3-second response times. The project involved deploying a fine-tuned open-source model that surpassed existing closed-source models, including GPT-5, in reasoning capabilities and speed. The development process was divided into several phases, starting with building a sophisticated knowledge retrieval system and evolving through a comprehensive evaluation framework that identified failure modes and refined prompts for consistency and accuracy. Genetic optimization and supervised fine-tuning were employed to create high-quality training data, with reinforcement learning further enhancing the model’s ability to perform precise retrievals and generate compliant responses. By eliminating vector embeddings in favor of grep-based text search, the system improved robustness and adaptability to document changes. After six weeks, the AI achieved a 95.7% compliance pass rate, significantly outpacing competitors in speed and accuracy, and set the foundation for scaling the system to support additional insurance products and more complex interactions.