We switched to a 5x cheaper LLM. Our costs went up.

Gitar evaluated the lower-cost Kimi K2.5 model as the primary orchestration model for its multi-agent pull-request workflow, which can make dozens or hundreds of LLM calls per change, but found that practical costs and reliability were worse than expected compared with Claude. An agent-loop bug occurred because Kimi could return populated tool calls alongside a `finish_reason` of “stop,” causing the system to skip tool execution and repeatedly retry with growing context until one session consumed about half a million tokens; resolving it required prioritizing response content over finish-reason metadata and normalizing provider-specific tool-call IDs. Even after these fixes, Kimi’s lower rate limits triggered frequent failovers to Claude, reducing provider-specific prompt-cache hit rates and creating costly cold-cache requests during long-running sessions. The experience led Gitar to conclude that nominal token prices and API-format compatibility do not reliably predict operational value, and that evaluations should use real workloads, account for provider behavior, failure modes, cache effects, and engineering overhead, and measure cost per successful outcome rather than price per token.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.