We Built an AI Agent to Investigate Production Issues

Stream developed a production-issue investigation agent for its backend team that analyzes Slack support messages and alerts, gathers evidence from logs, databases, metrics, and source code, and posts cited root-cause findings while leaving all production changes to engineers. The system uses a small model for triage, a larger model for investigation, and an independent fresh-context review session that attempts to disprove conclusions before results are delivered, marking findings as unverified when evidence is insufficient. Safety measures include isolated read-only sandboxes, restricted tools and networking, host-protected database credentials, filtered Slack output, and a service layer that controls any external action, with approved fixes becoming draft pull requests or engineer-executed runbooks. The team evaluates the agent against 22 cases based largely on real incidents and previous errors, including irrelevant messages, misleading requests, nonexistent-tool traps, customer-side problems, and known reasoning mistakes. Stream reports that the agent can shorten investigation time, such as identifying CPU-bound cache capacity during a 5xx incident, while maintaining human responsibility for decisions, remediation, and incident ownership.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.