Bitrise has embarked on integrating AI into its developer workflows to streamline and enhance processes from code commit to production release. A dedicated team focused on identifying and developing AI features that provide genuine value to developers, emphasizing the need for non-deterministic systems that automatically detect regressions. The team created an internal evaluation framework in Go to benchmark various AI coding agents, aiming to run tests in parallel and report results efficiently. After testing several agents, including Claude Code, Codex, Gemini, and OpenCode, Bitrise identified the need to develop an in-house coding agent to match Claude Code's performance while avoiding vendor lock-in and ensuring seamless integration into their ecosystem. This approach allows for greater autonomy, flexibility, and innovation, as well as the ability to embed verification directly into the workflow. Future posts will delve into the technical challenges and successes of scaling AI feature development across the company.