The Golden Metrics: Characters per Opportunity and Percentage Code Written

Characters per Opportunity (CPO) and Percentage Code Written (PCW) are proposed as definitive metrics to evaluate AI code assistants, with CPO focusing on benchmarking autocomplete capabilities and PCW assessing the actual value derived by developers using these tools. CPO measures the tool's effectiveness by evaluating suggestions without being subject to gameable factors, while PCW reflects the amount of code contributed by the assistant. Despite being conceptually related, CPO and PCW are not mathematically linked. Codeium uses these metrics to guide product development, boasting a CPO of 1.27 and a PCW of 44.6%, indicating that nearly half of the new code by its users is generated by the assistant. Autocomplete is emphasized as a critical feature due to its passive nature and the sheer volume of suggestions it can provide, significantly outweighing other features like chat in terms of value. The critique of competitors like SourceGraph Cody and TabNine highlights the importance of balancing acceptance rates with real value delivery, as optimizing solely for acceptance can lead to superficial improvements without enhancing the actual utility of the product. Codeium's focus remains on genuine product improvement and value delivery rather than chasing marketable metrics.