June 2026 Summaries

Surge AI's evaluation benchmarks, GDP.pdf and Riemann-bench, are designed to assess advanced AI models on complex, domain-specific tasks that reflect real-world challenges and require expert-level reasoning. GDP.pdf focuses on professional multimodal reasoning over documents in diverse sectors like finance and healthcare, testing models' abilities to parse and synthesize intricate information, while Riemann-bench evaluates frontier mathematical reasoning with problems sourced from Ivy League academics. These benchmarks, cited in Anthropic's Fable 5 and Mythos 5 release, highlight the importance of expert-built evaluations in distinguishing genuinely improving models from those only excelling at saturated tests. As straightforward benchmarks become less informative, the ability to measure sophisticated, expert-graded capabilities becomes crucial for accurately assessing AI progress.

ComplexConstraints is a benchmark designed to evaluate the capability of models to handle complex, entangled constraints that mimic real-world professional scenarios. Unlike simpler benchmarks that focus on explicit and independent constraints, ComplexConstraints incorporates conditional, planning, multi-step, negative, and implicit constraints, reflecting the nuanced demands of professional tasks such as film production, restaurant staffing, and office procurement. Each prompt involves numerous interdependent constraints, challenging models to maintain consistency and accuracy across various tasks. The benchmark has shown that training models on such complex constraints not only improves performance on the specific benchmark but also enhances generalization to other benchmarks like AdvancedIF and MultiChallenge. Results indicate that models trained on the ComplexConstraints data exhibit improved task completion rates, better adherence to constraints, and increased ability to retain and apply user preferences in multi-turn scenarios. These improvements suggest that mastering complex constraint handling could significantly enhance the practical utility of AI models in professional environments.

Microsoft conducted a study using Surge's human evaluation services to assess the real-world effectiveness of their AI model, MAI-Thinking-1, as compared to Claude Sonnet 4.6. Instead of relying solely on benchmarks, which can sometimes be misleading, they employed blind human evaluations to determine which model users preferred across various tasks. This method highlighted the importance of human preference data, demonstrating that while benchmarks are essential for measuring specific capabilities, they do not fully capture the user experience. The study revealed that MAI-Thinking-1 was favored over its competitor in a significant number of tasks, emphasizing the value of human judgment in evaluating AI performance. Surge AI offers such evaluations to other developers to ensure their models not only perform well on paper but also provide a satisfactory user experience.