Why your traces and evals belong in the same place

AI regressions often go unnoticed on dashboards because they typically manifest as incorrect outputs rather than latency or error rate issues, with the first indications usually emerging from support queues rather than the engineering stack. Teams that efficiently catch these regressions integrate evaluations and trace logs within the same tool, reducing the manual process and context-switching that typically delays fixes. When traces and evaluations are combined, suspicious traces can be quickly turned into datasets, and scoring functions can be reused to monitor fixes in production, significantly speeding up debugging processes. Automating the quality assurance process is the next step once traces and evaluations are centralized, as it allows the system to propose solutions, run evaluations, and even suggest new scoring functions for unaddressed patterns, leading to a more proactive approach in identifying and resolving issues. Centralizing these processes is crucial for enabling automation and gaining a comprehensive understanding of the system's performance, thus streamlining workflows and allowing many tasks to be completed automatically.