September 2026 Summaries

Measuring the ROI of AI coding tools is difficult because software productivity has no single reliable metric, and comparisons with a hypothetical no-AI baseline are vulnerable to self-reporting errors, selection bias, and changing organizational conditions. Rather than relying on lines of code, pull requests, stories, flow measures, or developer sentiment alone, the proposed approach compares several throughput measures per engineer against past performance and calculates both estimated ROI and the total cost per unit of work, including developer and AI-tool spending. An example shows how higher code output and pull-request volume can coexist with fewer completed stories, revealing shifts in work structure or bottlenecks outside coding rather than a definitive AI verdict. Quality indicators such as change failure rates, bug backlog, and maintenance work must also be monitored, since apparent speed gains may create later costs, while organizations should expect an initial productivity dip as teams learn tools and adapt processes. The Build framework groups assessment into AI leverage, engineering effectiveness, and return on engineering, encouraging teams to track multiple trends and business outcomes over time instead of presenting a single, potentially misleading ROI figure.

Taito.ai, an eight-person people operations startup with four engineers, uses an agent-centered software development workflow in which engineers rapidly create disposable end-to-end prototypes, promote only promising ideas to a team-reviewed implementation plan, build features largely through agents, and then divide work into pull requests mainly to focus human review on sensitive changes. Its agents handle feature coding, bug investigation, testing, CI failures, and automated reviews, while deterministic checks and review rules flag areas such as authentication, authorization, personal data, secrets, and complex migrations for human attention. The company reports that its Fixer agent reduced its backlog of unresolved reported bugs to zero and resolves about 70% of reported issues within 30 minutes, largely by addressing small problems that previously lacked priority. Taito treats repeated review feedback and coding conventions as maintained “skills” for agents, supported by extensive automated tests, sandboxed permissions, local Kubernetes environments, and CI pipelines. Humans remain responsible for architectural understanding, high-stakes trade-offs, security and customer-impact decisions, and close review of foundational systems, while the company acknowledges that its approach is easier for a small, young organization without substantial legacy software and that the appropriate balance between agent autonomy and human review remains unsettled.