Man in the Loop vs. LLM in the Loop

Yotam Luz, a Principal Data Scientist at Vonage, discusses the shift from human oversight to automation in AI, particularly in the context of Vonage AI's efforts to redesign speech-to-text (STT) systems using Large Language Models (LLMs). This shift is driven by the limitations of traditional benchmarking with human-generated "ground truth" and the need for scalable, unbiased, and context-aware evaluation methods. LLMs synthesize consensus transcriptions from multiple STT outputs, providing reliable reference transcriptions that allow for a fair comparison of model accuracy. The new pipeline demonstrates that LLM-generated references can deliver nearly identical Word Error Rates (WER) to human-labeled data, proving their robustness and scalability for benchmarking purposes. Despite higher error rates in human-labeled data, such references remain valuable for training, as demonstrated by the improved performance of models fine-tuned on this data. This approach accelerates benchmarking across new models and languages, eliminating the need for manual transcription while maintaining the benefits of human-labeled data for model development.