September 2026 Summaries
6 posts from Surge AI
Filter
Month:
Year:
Post Summaries
Back to Blog
GDP.xlsx is a benchmark of 70 real-world Excel tasks across 12 professional domains designed to test whether AI agents can navigate and reason over complex workbooks rather than merely extract cell values. It emphasizes that spreadsheet logic is often distributed across tabs, formulas, lookup tables, formatting, comments, hidden rows, and embedded business rules, requiring models to determine which sources are authoritative and how information connects. The leading model cited, Gemini 4 Argon, achieved 38.3%, illustrating the difficulty of the benchmark. Case studies describe models overlooking decisive information on separate tabs, such as manufacturer-specific certification rules and lease-break clauses, even when they correctly identified related data elsewhere. Built with domain professionals from actual workplace workflows and reviewed against detailed grading rubrics, the benchmark aims to assess whether an AI can provide decisions that users could trust without extensive human guidance or auditing.
Sep 30, 2026
1,820 words in the original blog post.
Surge AI post-trained Kimi K2.7 on 1,465 professional-document tasks from its GDP.pdf dataset and reported substantial improvements both on held-out PDF tasks and on GDPval, a separate benchmark requiring workspace exploration, tool use, calculations, and finished business deliverables. Strict success on GDP.pdf increased from 11.0% to 24.5%, while GDPval tasks scoring at least 90% rose from 19.5% to 32.7% and perfect scores from 2.7% to 5.9%. The reported gains extended to tasks without PDFs, suggesting that training improved a broader habit of identifying all relevant evidence before acting rather than merely improving document reading. Comparisons of model trajectories found that the trained model accessed more relevant sources while reading less total text, repeated itself less, and more consistently accounted for complete records, dependencies, and task requirements. Examples included combining all weekly timekeeping sheets before preparing monthly reports, carrying every checklist item into an action tracker, and updating schedules when delays affected completion dates. The authors conclude that realistic, complex PDF tasks can strengthen transferable professional-reasoning skills, although the model already possessed the tools and file-handling abilities needed for the evaluation environment.
Sep 29, 2026
2,680 words in the original blog post.
DAYJOB is a benchmark suite introduced to assess whether AI agents can perform realistic, long-horizon professional knowledge work in healthcare and finance, where tasks are often ambiguous, spread across many documents, and require judgment rather than simple execution. Its creators argue that existing benchmarks such as GDPval frequently provide overly detailed instructions, limited inputs, and tasks focused on formatting or routine document production, while DAYJOB uses shorter requests, roughly 20–26 supporting files per assignment, and assignments estimated to take professionals about 20 hours. The initial release includes 130 expert-designed tasks covering areas such as clinical decision support, patient safety, investing, banking, and corporate finance, with task-specific evaluation rubrics. Examples show agents missing a major currency-unit pricing error that overstated a fund position by $25.7 million and failing to challenge a potentially unsafe Bell’s palsy discharge plan despite concerning symptoms. The reported top model scores are below 25% in both healthcare and finance, suggesting that although current agents can handle well-specified calculations and writing tasks, they remain limited in independently identifying relevant information, resolving ambiguity, and making consequential professional judgments.
Sep 23, 2026
2,790 words in the original blog post.
Surge AI describes its role in Anthropic’s research on automated alignment researchers, which used Claude-based agents to search literature, propose safety interventions, train and evaluate models, and iteratively improve methods across ten alignment problems such as deception, sycophancy, jailbreaks, privacy violations, and reward hacking. Surge built and operated the human comparison baseline, recruiting 28 experienced AI safety researchers who each had up to eight hours to submit a single proposal for selected failures, while also managing structured submissions, quality control, and expert review. Anthropic reported that its automated researchers produced stronger methods than the human-proposed baselines on all seven failures with human comparisons, though it noted that agents had the advantage of repeated iteration. According to the account, successful methods improved safety metrics without substantially reducing general capabilities and showed transfer to held-out benchmarks, open-ended audits, and models up to 4.7 times larger, illustrating a potential role for automated systems in complementing human alignment research.
Sep 15, 2026
1,098 words in the original blog post.
Surge AI reports that reinforcement-learning-only post-training of Kimi K2.7 on 1,700 coding tasks improved its Pass@1 performance across five external coding benchmarks by 4.7 to 20 percentage points, with gains transferring across three different agent harnesses and, in some cases, surpassing reported results from larger models. Analysis of paired trajectories suggests the trained model became more effective not by taking more steps, but by reducing median trajectory lengths while improving four software-engineering behaviors: retaining every specification detail, testing requirements rather than only supported implementation paths, preserving existing functionality against regressions, and constructing independent checks when no reference answer is available. The training data combined repository-change and terminal-deliverable tasks with hidden checks, while a reward function gave partial credit for completed requested behavior but assigned zero reward if existing tests regressed. Although the authors say they have not conducted ablations proving the reward design caused the changes, they argue that the results show high-quality RL environments can teach transferable engineering habits, helping a model move from solving the core task to reliably delivering complete, shippable software.
Sep 11, 2026
3,100 words in the original blog post.
Surge AI’s comparison of recent frontier-model releases finds that Fable 5.1 leads overall on its Tuesday Work Index with a score of 68.7 and tops benchmarks for professional chart reasoning, policy-document instruction following, and enterprise agent performance, though its cost efficiency varies by task. Muse Spark 1.3 records the largest generation-to-generation index gain and leads ComplexConstraints, where it combines strong performance on interacting professional requirements with favorable costs, while showing uneven results on chart reasoning and advanced mathematics. Gemini 3.8 Flash reaches 61.1 on the index and is highlighted for cost-efficient hard reasoning, particularly after a 12-point improvement on Riemann-bench, although higher reasoning settings do not consistently improve its results across workloads. Together, the releases suggest that top-end capability continues to advance while lower-cost models narrow the gap, and GPT-6 Astra remains absent from the comparison pending completion of its evaluation.
Sep 10, 2026
1,631 words in the original blog post.