How to curate observability data for AI agents

Building a successful debugging agent for Multiplayer involved overcoming initial challenges with observability data by implementing a structured data curation process. Initially, the agent struggled with raw data, leading to irrelevant actions and ineffective fixes due to the overwhelming signal-to-noise ratio. To address this, Multiplayer developed a multi-stage process to transform the data into a structured and context-rich package. This process includes aggressive grouping and correlation of events, assessing the fixability of issues, adding release context and metadata, and reformatting data for machine consumption. By focusing on what the agent needs to understand to produce effective fixes, the curated approach significantly improved the agent's performance, enabling it to generate more accurate and reliable solutions. This shift from raw data exposure to curated data preparation was critical in harnessing the potential of AI for debugging, ensuring that the agent could focus on relevant issues and generate fixes that hold up in production.