Agent memory benchmarks are baked

Agent memory benchmark scores depend heavily on methodological choices such as the answering and judging models, prompts, scoring rules, retrieval budgets, and whether difficult questions or partial credit are included, making headline figures difficult to compare without detailed disclosures. SurrealDB’s analysis of BEAM and LoCoMo benchmark runs argues that some published approaches make results easier by exposing answer-related metadata or question categories to models, using lenient scoring, expanding retrieval context, excluding adversarial questions, or relying on permissive judges. Its own BEAM 1M run reported a 77.2% mean score and a 57.1% strict score, which counts only answers receiving full credit, while using Kendall tau for event-ordering questions and publishing its 700 individual answers and rubrics. On LoCoMo, SurrealDB reported a 75.8% strict score across all categories, alongside higher scores under Mem0’s more lenient protocol. The company recommends evaluating benchmark claims by examining the answering and judging models, prompt information, treatment of partial credit, retrieval context, and availability of per-question outputs.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.