October 2026 Summaries

Comet reported that an unauthorized party used an API key from an inactive, non-administrative comet.com account between October 24 and 26, 2026, to access a small number of internal Opik test workspaces and exploit a flaw in AI provider settings, obtaining 29 internally used OpenAI and Anthropic API keys. Comet stated that no customer data, customer accounts, customer AI provider keys, or internal systems were accessed, and that the exposed third-party keys were revoked. After an AI provider identified unusual activity on October 26, Comet blocked the vulnerable functionality, deployed a fix, removed the inactive account, audited workspace membership and logs, rotated internal credentials, and experienced roughly two hours of Opik downtime while some features were temporarily disabled. The company also added provider-address verification and restrictions, along with enhanced monitoring, and said cloud users need take no action, while self-hosted users were not affected but are advised to upgrade because the underlying flaw existed in earlier open-source versions.

Opik has introduced generative dashboards, a feature powered by its conversational AI assistant, Ollie, that lets users create custom monitoring widgets for AI agents through plain-language requests rather than manual SQL queries and chart design. Ollie translates requests into read-only SQL queries against ClickHouse data, analyzes the results, and proposes Vega-Lite visualizations that can be added, resized, shared, and later refined through further conversation. The feature is intended to help teams track organization-specific metrics across trace, span, thread, experiment, and feedback-score data. In an automotive example, a team uses Ollie to compare hallucination scores for vehicle-specification responses across agent versions and then converts the comparison into time-series trendlines, allowing it to identify declining accuracy, investigate relevant traces, and potentially fix or roll back problematic releases. Generative dashboards are generally available in Opik for users with free or existing accounts.

Decision models return calibrated probability distributions over predefined answer options, distinguishing them from structured-output LLMs that generate tokens sequentially and expose next-token likelihoods rather than confidence that an answer is correct. They resemble flexible, modern classifier systems because they can receive new questions and answer choices at inference time, making them suitable for routing, scoring, gating, classification, and other bounded decisions where confidence thresholds can reflect the costs of mistakes. Using a Qwen-based prototype trained on the typed-decisions benchmark, the article shows how multiple independent questions can be packed with one shared input into a single transformer pass through a custom attention mask, reducing processing time while preserving outputs. Its experiments find that training against full target probability distributions with cross-entropy, rather than conventional single-label fine-tuning, is the main source of well-calibrated predictions: both a dedicated decision head and a language-model-head readout achieved similar test accuracy of about 0.78 and low calibration error near 0.02, while the standard fine-tuned baseline was substantially overconfident. Evaluation tracked in Comet and performed in Opik also confirmed question isolation and limited sensitivity to irrelevant options, although reversing answer-option order still altered some trained-model predictions, highlighting an unresolved robustness issue.