August 2026 Summaries

Benchmark results on 113 DeepSWE software-engineering tasks indicate that GLM-5.3 Flash is substantially cheaper and faster than the full GLM-5.3 model while retaining much of its capability, though with lower single-attempt reliability. The full model achieved 69.0% pass@1 versus Flash’s 63.4%, but the gap narrowed from 5.6 points to 2.6 points after four attempts, suggesting that Flash’s main loss from distillation is consistency rather than its ability to solve tasks. At $0.24 per rollout compared with $3.99, Flash delivered 264 solves per $100 versus 17 for the full model, completed runs faster, and performed comparatively well in concurrency, Python, data modeling, and protocol tasks, while the full model remained stronger in JavaScript, query-oriented work, and several reasoning-heavy languages. The analysis found that Flash retained solutions for 93 of the 99 tasks solved at least once by the full model, but was less able to turn longer attempts into successes and more likely to break previously passing baseline tests, with a 6.9% regression rate versus 4.4%. It recommends using Flash by default in cost-sensitive or retry-tolerant workflows, validating its changes with regression tests, and escalating rejected outputs to the full model, a cascade reported to reach 80.9% accuracy at $1.70 per task.

A DeepSWE v1.1 comparison across 113 software-engineering tasks and four trials per model found GLM-5.3 and Claude Fable 5 statistically close on first-attempt accuracy, with Fable at 69.7% pass@1 and GLM at 69.0%, but GLM led on repeated attempts, reaching 81.1% versus 77.1% at pass@2 and 87.6% versus 84.1% at pass@4. GLM’s reported rollout cost was $3.99 compared with Fable’s $21.63, producing an estimated 17 versus 3 solved tasks per $100, while average completion times were similar at roughly 35 minutes. GLM showed broader task coverage and strengths in JavaScript, TypeScript, concurrency, query/configuration work, runtime internals, and program analysis, whereas Fable performed better in Rust, Python, data modeling, serialization, build and operations, and protocol-conformance tasks. Both models had similarly low regression-related failure rates and a high per-task outcome correlation of 0.65, meaning that using both together added limited coverage; the analysis therefore recommends GLM-5.3 as a lower-cost default, with Fable reserved for Rust-heavy or serialization-critical work, while noting scoring choices, infrastructure errors, and incomplete public trajectory data as caveats.

A comparison of GLM-5.3 and GPT-5.6 Sol on 113 DeepSWE software-engineering tasks across four trials each finds that Sol leads on first-attempt success, reliability, and speed, achieving 72.7% pass@1 versus GLM-5.3’s 69.0% while averaging 19 rather than 35 minutes per rollout. GLM-5.3 costs less than half as much per rollout at $3.99 versus $8.37, delivers more estimated solves per $100, and overtakes Sol when retries are allowed, tying at pass@2 and leading 87.6% to 85.8% at pass@4. The models have distinct strengths: GLM-5.3 performs particularly well on JavaScript, Rust, query and configuration languages, runtime internals, and stateful reactivity, while Sol leads in Python, Go, TypeScript, protocol conformance, and several systems-oriented domains. Sol’s failures more often introduce regressions in existing tests, whereas GLM-5.3 more frequently produces near misses without breaking the baseline suite. Because their task-level outcomes show limited correlation and together cover 106 of 113 tasks, the analysis recommends a verifier-gated cascade that tries GLM-5.3 first and escalates failures to Sol, estimating 85.9% task coverage at $6.61 per solved task, though the findings are limited to the stated DeepSWE records, scoring rules, and assumptions about independent attempts and test-based verification.

A comparison on 113 DeepSWE software-engineering tasks, using four trials per model, found that GPT-5.6 Sol delivered stronger single-attempt performance than DeepSeek V4 Pro 0813, with 72.7% versus 62.8% pass@1, faster median completion times, fewer steps, and higher per-task reliability. DeepSeek V4 Pro 0813, however, cost $0.24 per rollout compared with Sol’s $8.37, achieved a higher pass@4 rate of 88.5% versus 85.8%, and provided far more solved tasks per dollar, making it more suitable for high-volume or retry-tolerant workflows. Sol led across most task domains and programming languages, particularly Python and Go, while Pro slightly outperformed it in Rust and stateful reactivity. Their failures also differed: Sol was more likely to introduce regressions into previously passing tests, while Pro more often produced near misses without breaking the existing suite. The reported optimal deployment strategy is a test-gated cascade that runs Pro first and escalates failed outputs to Sol, achieving 83.0% task resolution at an average cost of $3.35 per task, outperforming either model alone on the combined accuracy-cost measure.

Together AI’s endpoint-level A/B testing feature lets teams divide live inference traffic among one control deployment and up to 20 variants using fixed percentage cohorts, enabling measurement of real product outcomes such as user ratings, retention, and task completion without application-side routing logic. Experiments route traffic only through the control’s base traffic share, require variants to have zero endpoint traffic weight, and use percentages that remain independent of replica counts and autoscaling. Teams can begin with low exposure such as 95/5, ramp variants by replacing the experiment’s member configuration through etag-protected updates, and run multi-way tests so long as percentages total 100% and exactly one control is present. Platform metrics are available by deployment, while response metadata enables product analytics to attribute quality outcomes to each cohort. If a variant wins, it can be promoted through a blue-green rollout before the experiment is deleted; if it loses or degrades, removing it from the experiment or deleting the experiment returns traffic to the control. Requests can be assigned consistently using a stable sampling key, while key-less requests are sampled per request. In a live demonstration, observed traffic closely matched configured 95/5, 80/20, and 50/50 splits, and deleting the experiment restored all traffic to the control after propagation.

Benchmark results on 113 DeepSWE software-engineering tasks show that Claude Fable 5 has stronger first-attempt accuracy than DeepSeek V4 Pro 0813, scoring 69.7% versus 62.8% pass@1, but costs about 90 times more per rollout at $21.63 compared with $0.24. With repeated attempts, Pro matches Fable at pass@2 and leads at pass@4, while delivering vastly more solved tasks per dollar and producing failures that are more often near-correct rather than major misses. Fable performs particularly well on Rust, serialization, data modeling, and other exact-contract tasks, whereas Pro leads in TypeScript, stateful reactivity, and concurrency and durability. Because the models succeed on substantially different tasks, with a low per-task correlation of 0.39 and combined coverage of 107 of 113 tasks, the analysis recommends running Pro first and escalating failed, test-verified results to Fable. This cascade reportedly reaches 82.7% accuracy at $8.28 per solved task, exceeding Fable alone’s accuracy while costing less than half as much per task.

Benchmark results on 113 DeepSWE software-engineering tasks find that GPT-5.6 Luna is the stronger standalone model, achieving 67.2% pass@1 versus DeepSeek-V4 Flash 0731’s 53.3%, leading across seven of eight task domains, all tested programming languages, and completing work faster. DeepSeek costs about $0.10 per rollout compared with Luna’s $0.61, however, yielding substantially more solves per dollar and causing regressions in existing test suites less often when it fails. Its relative strengths are query and configuration work and more competitive Rust performance, while JavaScript and reasoning-intensive tasks show large deficits. Despite limited complementary task coverage, using DeepSeek as a first-stage model and escalating failed attempts to Luna produces a reported 78.9% success rate at $0.385 per task, exceeding Luna alone in both accuracy and cost efficiency.

Kimi K3, developed by Moonshot AI, is a groundbreaking open-source model with 2.8 trillion parameters, setting a new standard in open-weight models by being the first in the 3-trillion-parameter class. It is designed for advanced tasks such as long-horizon coding, deep reasoning, and end-to-end knowledge work, making it competitive with proprietary models like Claude Fable 5 and GPT 5.6 Sol. The model features innovations such as Kimi Delta Attention and Attention Residuals, enhancing its ability to process long sequences efficiently. Kimi K3 supports a 1M token context window and is available through Together AI, where it offers flexible reasoning capabilities and competitive pricing. The model excels in several coding and agentic benchmarks, demonstrating its frontier-level performance while also supporting vision tasks with multiple image inputs. By utilizing tools like automatic caching and dynamic tool loading, Kimi K3 provides a scalable and efficient solution for complex computational tasks, making it accessible for production use with a single API call on Together AI.