OpenAI GPT-6 Sol: An Evaluation of Code Quality and Security

Sonar’s evaluation of GPT-6 Sol on 4,444 Java tasks from HumanEval, MBPP, and ComplexCodeEval found that it passed 83.09% of 544 test-backed tasks, below GPT-6 Astra’s 85.85% but above earlier GPT-5 variants. Sol generated 480,125 lines of code, 26.9% less than Astra, and produced 21.8% fewer total findings, including fewer bugs and code smells in absolute terms, although its issue densities per line were generally higher. Security was the main exception: Sol produced 137 vulnerabilities versus Astra’s 117, raising vulnerability density by 60%, with cryptography misconfiguration and insecure resource handling accounting for most findings; blocker vulnerabilities remained rare, with one found in Sol’s output. Sol’s code had slightly higher cyclomatic and cognitive complexity densities, substantially fewer comments, and modestly smaller functions, while improving notably in concurrency and exception-handling bug findings but increasing rates in several other bug categories. Although Sol used fewer total tokens, it nearly doubled reasoning-token use while generating less code, suggesting a “more thinking, less writing” profile. Overall, Astra offered a higher functional pass rate and cleaner per-line quality, while Sol reduced code volume and the total review workload, making the choice dependent on whether teams prioritize fewer artifacts to inspect or stronger correctness and security density.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.