Building a self-improving agent on a context graph of human disagreement

A self-improving AI agent can be developed by leveraging existing data from human corrections, captured as a context graph, without the need for retraining. This approach addresses the discrepancy often found between AI recommendations and human decisions based on institutional knowledge not documented in formal policies. By capturing and analyzing human overrides in a structured context graph, patterns emerge that can be fed back into the AI system to improve its decision-making process. A procurement agent demo illustrates this, where human reviewer Vera Fye's overrides were used to enhance the agent's accuracy from a 53.8% to an 83.1% match rate over four cycles, solely by integrating structured human feedback. This method shows promise in various applications where human judgment complements AI, enabling continuous improvement without altering the agent's source code. The broader implication is the potential to harness human-AI disagreement as a valuable signal rather than discarding it as noise, thus enhancing agent performance in alignment with real-world complexities and institutional knowledge.