AssemblyAI: A voice agent can fail without throwing an error

At AssemblyAI, Griffin Sharp, an applied AI engineer, demonstrated a method to automatically identify and diagnose issues in voice agents, which are used in various applications from drive-thru ordering to triaging support calls. Despite manual testing before deployment, these voice agents often encounter real-world issues like noisy audio and crosstalk that aren't easily caught early on. The current feedback loop relies heavily on customer reports, and even with good logs and transcripts, diagnosing issues can be complex due to the subtle nature of errors. To address this, AssemblyAI developed a pipeline that processes calls post-factum using a series of automated steps: re-transcribing calls with a more reliable asynchronous API, summarizing the calls through an LLM Gateway, diagnosing patterns of failure, and synthesizing these insights across multiple calls to identify fleet-wide issues. In a demo involving 50 calls, this process revealed a deployment-related issue that increased endpointing errors due to a configuration setting. Although the pipeline provides a strong diagnostic foundation by generating hypotheses and highlighting failure rates, final verification and adjustments still require human intervention to confirm and address these issues across different conditions.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.