AI Configs introduces a novel approach to measuring the quality of AI systems in real-time through online evaluations, addressing the challenges posed by the nuanced nature of AI system behavior, which traditional software testing methods cannot adequately capture. By integrating this capability into the same control plane used for managing releases and experiments, AI Configs allows teams to continuously monitor and assess the performance of AI systems using metrics such as accuracy, relevancy, and toxicity. This real-time evaluation is facilitated by LLM-as-a-Judge, which automatically scores AI outputs to ensure quality standards are met and to guide decision-making during rollouts and experiments. AI Configs enables teams to make evidence-based decisions by comparing configuration variants and setting quality thresholds that trigger automatic adjustments if necessary. This system transforms quality measurement from a reactive process into a proactive and ongoing feedback loop, enhancing the ability to maintain high standards in AI performance and user experience. Currently available in early access, AI Configs offers tools for quality measurement that allow teams to address issues like tone drift and context loss, fostering a continuous learning environment to optimize AI outputs.