How to build a software factory that tests its own prompts

An AI-driven software factory is described as a pipeline that converts GitHub issues into released code through specialized agents for specification, implementation, testing, review, routing, human approval, and release, while retaining deterministic control over repository actions through conventional Python code. Built around a trading-card application, the system traces every issue as a single Arize AX observability record, linking agent prompts, tool activity, GitHub events, CI outcomes, reviewer decisions, human gate verdicts, reverts, and releases. Its central premise is that as agents make coding faster, reliable evaluation becomes the main constraint, especially because agents can miss defects, generate false alarms, or optimize against weak tests. Human review-gate outcomes are harvested as labels to evaluate reviewer agents, routing decisions, and LLM-based judges, while prompt changes are tested on recorded pull requests and promoted only if they reduce misses without increasing false alarms. Experiments found that an initial reviewer-prompt change failed because the agent identified issues but treated them as out of scope, whereas a later prompt that emphasized specification goals, branch coverage, and formal findings eliminated known misses in repeated in-sample tests. The author also improved an overly cautious routing agent, cautions that both LLM judges and human gate decisions can be imperfect, and argues that factories should collect and connect feedback signals from their earliest stages to support ongoing, evidence-based agent improvement.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.