Building Investigations: what it takes to build an AI SRE

Incident.io’s two-year effort to build Investigations, an AI SRE that analyzes telemetry, deployments, code, documentation, and prior incidents during outages, illustrates the large gap between an impressive AI prototype and a dependable operational product. The company emphasizes rigorous backtesting on frozen historical incidents to measure accuracy without leaking future information, alongside a modular agent harness that gathers evidence, creates and revises structured findings, generates hypotheses, and uses adversarial checks to challenge conclusions. Making the system useful also required controlling noise through confidence, novelty, timing, tone, and evidence requirements for incident-channel updates, since inaccurate or excessive messages can quickly erode user trust. At scale, the product must manage token costs, telemetry-query load, security, varied customer environments, and stale organizational knowledge, using continuously curated memories and a living production-environment model called Nexus. The author argues that becoming an AI company involves substantial systems work around models, including evaluation, observability, self-updating knowledge, and safeguards, and advises teams considering building their own AI SRE to account for this ongoing complexity and maintenance burden.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.