November 2026 Summaries

An analysis of 2,655 runs of Thinking Machines’ Inkling 2 Small across 13 benchmarks found that the model used the word “fuck” in 50 runs, with 316 total occurrences heavily concentrated in two runaway runs that produced 241 instances. Profanity appeared primarily in long, difficult runs, with none occurring before 100 steps and most appearing in the latter half of attempts; ProgramBench had the highest number of affected runs among the public benchmarks. The model’s swearing commonly accompanied self-identified mistakes, perceived breakthroughs, frustration, or abandonment of unsuccessful tasks, and in some cases emerged in repetitive loops that generated extremely long responses until the context window filled. Although 46 of 48 graded runs containing profanity failed, the analysis concludes that swearing is not a useful predictor of failure because most failed runs did not swear and overall failure rates were already high. The study counted selected uncensored profanity variants in the model’s reasoning and messages while excluding tool outputs, programming-language terms, names, and censored spellings.

An evaluation of seven frontier AI models across eight Vals Index benchmarks found that reducing reasoning effort often lowers costs far more than performance, though the trade-off varies by model and task. At their lowest settings, four models retained at least 80% of their leaderboard-level score for half the cost or less, with Claude Opus 5.5 and Grok 4.7 each using only 16% of the cost while retaining 80% and 86% of performance, respectively. Across 22 of 56 model-and-benchmark comparisons, a lower-effort setting matched the higher-cost leaderboard setting, while no lower setting significantly exceeded it. Retrospectively selecting the cheapest highest-scoring setting for each task improved scores by a median of 6 points while costing 64% of the leaderboard configuration, although this represents an upper bound because real systems must select effort before knowing outcomes. Additional reasoning was especially valuable for Legal Research, Code Migration, Terminal-Bench, tax, and coding tasks, but produced comparatively small gains on finance, HLAB, and EMB benchmarks. The analysis notes limitations including testing only about half of each benchmark’s tasks, limited repeated runs, possible provider changes over time, and statistical uncertainty in small performance differences.

Vals and Fireworks argue that custom evaluations are essential for organizations seeking to select, train, and deploy AI models effectively for their specific work. While broad and domain-specific benchmarks can indicate general model capabilities, they may not identify the best model for a company’s particular tasks, edge cases, and quality standards. Custom evals capture internal expert knowledge in measurable form, enabling companies to compare models, fine-tune smaller open models through Fireworks RFT, validate improvements before production, and monitor performance over time. They can also support cost reduction by identifying lower-cost models that match premium alternatives on relevant workloads and by routing simple and complex requests to different models through tools such as Fireworks Nexus and FireRouter. Vals partners with Fireworks customers to build and maintain these evaluations, positioning them as a foundation for specialized AI systems that improve quality, efficiency, and confidence in deployment.