41 Experiments in 6 Days: What the Human Was Actually For

A six-day project used Claude Code to run 41 experiments on long-term conversational memory retrieval, increasing LoCoMo benchmark F1 from 0.392 to 0.666, though the authors emphasize that the main lesson was not the score but the importance of human oversight in evaluating objectives. The agent wrote nearly all code, implemented hypotheses, and uncovered several technical bugs, while humans provided the critical checks that exposed misleading evaluation signals, including category hints that leaked answer formats, a recall metric unable to count extracted facts, and fact extraction that added cost and search noise without improving F1. Fast five-to-seven-minute evaluation cycles, a constrained TypeScript harness, and separating editable experiment code from fixed evaluation infrastructure enabled rapid iteration, but measured run-to-run F1 variance of about 0.04 required caution when interpreting small changes. The final, simpler system removed several components, made image captions searchable, combined grep with ranked search, improved speaker-based filtering, and expanded retrieval context. Using Postgres as a unified store for semantic, full-text, and grep retrieval made schema and query changes inexpensive, supporting experimentation that would have been more complex across separate vector, search, and relational systems.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.