Does your LLM thing work? (& how we use promptfoo)

The text provides insights into the experiences and lessons learned by the AI team at Semgrep while implementing and evaluating Large Language Model (LLM) features. The team categorizes quality metrics into Behavior, Feedback, and Laboratory, each with its own challenges and benefits. Behavior metrics focus on the intended impact of AI features, while Feedback metrics involve user-generated ratings that can be biased or require segmentation for accuracy. Laboratory metrics, which include reproducible test suites and team evaluations, offer a shorter feedback loop but demand significant infrastructure. The document discusses the complexities of setting up a testing system that mirrors production environments, emphasizing the need for immutable and serializable template variables and the avoidance of dynamic rendering. It introduces the promptfoo tool, which facilitates LLM testing by managing prompts, template variables, and providers, and highlights its benefits over proprietary systems. The team employs various strategies to gather template variables, including using a staging database for simpler features and capturing real-world data for more complex scenarios. The system allows for the rapid evaluation of model variations and prompt adjustments, aiding in confident decision-making for model upgrades and feature enhancements.