October 2025 Summaries

Implicit caching in large language models, such as those used by OpenRouter, has sparked debate over whether it constitutes data retention, particularly in light of Google's stance that it does. Implicit caching involves automatically maintaining a cache of key and value tensors that represent a model's intermediate activations, enhancing performance by reducing computational redundancy without explicit user control. This process typically uses NVMe/SSD storage for efficiency, but only ephemeral KV tensors are stored, which are difficult to reverse-engineer into the original prompt data. Despite Google's conservative interpretation of implicit caching as disqualifying from Zero Data Retention (ZDR) compliance, OpenRouter argues that this form of caching does not equate to data retention since the data is transient and not operationally extractable. OpenRouter maintains transparency about which endpoints use implicit caching and continues to work with providers to safeguard sensitive data while ensuring that customers can make informed decisions about their data protection preferences.

Implicit caching in large language models (LLMs) refers to the automatic storage of key and value tensors, which represent intermediate activations, to enhance computational efficiency without explicit user control. This process, particularly when employing SSD storage, allows providers to handle more concurrent sessions by skipping prompt prefill, thus saving computational resources and costs. While Google views implicit caching as data retention, most providers and OpenRouter consider it a performance optimization, not a true retention mechanism, as the cached data is ephemeral and not aligned with raw tokens or user data. OpenRouter monitors and documents endpoint practices to ensure compliance with Zero Data Retention (ZDR) standards, emphasizing transparency and allowing users to configure routing based on their preferences or concerns. Despite the potential for a sophisticated actor to theoretically recover information from caches, the data is typically encrypted and not retained in a meaningful or operational way, aligning with the ZDR principles for protecting user data.

OpenRouter has introduced Exacto, a new set of endpoints designed to enhance tool-calling accuracy by utilizing providers with proven success rates. This initiative responds to observed variances in model performance across different providers, despite running identical models. OpenRouter maintains strong relationships with its providers, ensuring they prioritize quality, although occasional discrepancies occur due to the complex nature of hosting inference models. By collaborating with Artificial Analysis, they have benchmarked models such as DeepSeek 3.1, revealing performance improvements as providers optimize their systems. Exacto endpoints leverage tool-calling data, provider preferences, and benchmarks to route requests to top-performing providers, aiming to reduce tool-calling failures and increase reliability. While Exacto is focused on tool-calling accuracy, OpenRouter plans to share more data publicly to aid user decision-making and help providers improve their performance. The new Exacto endpoints are expected to gain popularity in agentic workflows, and OpenRouter is committed to refining and expanding its offerings based on feedback and ongoing analysis.

OpenRouter introduces "Exacto," a new set of endpoints designed to enhance tool-calling accuracy by selecting providers with higher success rates. These endpoints aim to address the observed discrepancies in performance among different providers using identical models, which arise due to the complexities of implementing models as production-grade inference. OpenRouter's unique position, handling billions of requests, allows it to identify and address these variances. By working closely with providers and conducting comprehensive benchmarks, OpenRouter has created an environment where tool-calling accuracy is significantly improved. Exacto endpoints are available for several models, including Kimi K2 and DeepSeek v3.1, and are expected to be popular in workflows that rely on precise tool usage. OpenRouter plans to continue refining this approach and share more data publicly to help users make informed decisions, while encouraging providers to improve and meet the criteria for inclusion in the Exacto routing pool.

Starting October 1st, OpenRouter is offering all customers 1,000,000 free "Bring Your Own Key" (BYOK) requests per month, automatically applied to existing users. After exceeding this limit, requests will incur a standard 5% fee. BYOK allows users to process requests using their own provider API keys, enabling billing by the provider rather than OpenRouter, and supports over 60 inference providers. Users can choose to fail over to OpenRouter if their keys run out of capacity. The initiative is designed to benefit startups and enterprises by allowing them to leverage existing provider credits and negotiated rates while accessing OpenRouter's features like failover protection, additional capacity, and spend controls. This offer applies to both individual users and organizations, with usage managed at the organization level for those with multiple users. The free tier includes all standard BYOK features without any changes to rate limits or features, and non-BYOK pricing remains unaffected.

Starting October 1st, OpenRouter offers its users 1,000,000 free "Bring Your Own Key" (BYOK) requests per month, allowing customers to use their provider API keys for requests and be billed by the provider instead of OpenRouter. This update is automatically applied to all BYOK users, and for those exceeding the 1 million requests, a standard fee of 5% is charged. BYOK, supported by over 60 inference providers, offers benefits such as failover protection, additional capacity, spend controls, and access to a broader range of models. The free tier maintains all standard features, including failover protection and spend controls, without affecting non-BYOK pricing, and resets monthly. Eligibility extends to all OpenRouter users and organizations, with better limits for organizations, while individual users and organizations are both eligible for the free tier.