November 2025 Summaries

Comet actively engages with the global machine learning community by hosting and sponsoring a variety of events that promote dialogue and innovation in AI. These events include renowned conferences such as NeurIPS and ICML, hackathons, summits, and meetups, often in collaboration with notable partners like AWS SageMaker, OpenAI, and GitHub. Spanning across locations like Las Vegas, Warsaw, and New York City, these gatherings offer opportunities for professionals to network, share insights, and explore advancements in AI technology. Notable events include the AI Tinkerers Meetup, the AWS Summit in London, and webinars focused on building scalable GenAI platforms, illustrating Comet's commitment to fostering a dynamic and interconnected AI ecosystem.

Comet's AI observability platform, Opik, has been recognized in Gartner's Innovation Guide for Generative AI Engineering, highlighting its increasing adoption by enterprises, startups, and developers for logging, testing, scoring, monitoring, and optimizing large language model (LLM) applications. Opik facilitates understanding and governance of generative AI models in production, enabling organizations to track model behavior, address operational drift, ensure compliance, and optimize performance throughout the lifecycle of AI deployments. As AI systems become integral to digital transformation efforts, the necessity of observability is underscored, and this acknowledgment by Gartner energizes Comet to continue innovating in areas such as observability, trust, governance, and automated optimization. The recognition is viewed as a milestone in Comet's journey to enhance value delivery and expand the potential of observability and optimization for AI agents. Gartner clarifies that its publications reflect its organization's opinions and do not constitute endorsements or factual statements.

In July 2025, an incident involving an AI coding assistant deleting a live company database highlighted the challenges of deploying large language model (LLM) applications, which can fail unpredictably due to their nondeterministic nature. This guide emphasizes the importance of adapting software testing strategies for LLM applications, which differ from traditional model evaluation and require a layered testing approach: unit, functional, regression, and production monitoring. Building a robust test dataset involves using production data, domain expert input, synthetic generation, and adversarial examples to cover core functionality and edge cases. Effective LLM testing combines evaluation methods like semantic similarity, LLM-as-a-judge, and rule-based checks to ensure reliability and safety. The guide also outlines common failure modes such as hallucinations, prompt injection, PII leakage, tone drift, and refusal errors, and provides best practices for systematic LLM testing, including integrating it with CI/CD processes and ensuring continuous improvement by incorporating real production failures into the test suite.

Agentic AI represents the next evolution in artificial intelligence, moving beyond predictive and generative AI to systems that autonomously act to achieve goals across various industries. This shift focuses on task completion rather than mere content creation, transitioning from passive responses to active executions. The concept of "agentic AI" is often misunderstood and overused, leading to skepticism within the tech community. True agentic systems operate on a spectrum, from simple applications to sophisticated multi-agent systems that autonomously manage complex tasks. The distinction between workflows and agents lies in control flow—workflows are deterministic, whereas agents exhibit dynamic control, enabling adaptability at the cost of predictability. Developers seek systems that are goal-oriented, autonomous, and capable of interacting with their environment. Despite industry cynicism, genuinely autonomous systems are being developed, and the key to successful implementation lies in understanding the principles of agency, building components such as reasoning engines, perception, action, memory, and orchestration, and addressing production challenges like reliability, cost, and security. The article highlights the potential of agentic AI in real-world applications, such as automated travel planning and enterprise operations, while acknowledging the hurdles of deploying AI agents at scale, emphasizing the importance of tools like Opik for building reliable, cost-effective, and secure agentic systems.

Opik's latest updates enhance the scalability and efficiency of LLM applications by integrating real-time alerts, advanced AI-powered prompt tools, and no-code experimentation in the Playground, fostering faster feedback, deeper insights, and improved reproducibility across workflows. Real-time production alerts now allow users to receive notifications for key events like errors and prompt changes, integrating seamlessly with existing CI/CD pipelines and communication tools such as Slack and PagerDuty. The AI-powered Prompt Generator and Improver facilitate prompt design and refinement without requiring prior expertise, while the no-code Playground enables comprehensive experimentation and analysis. Additionally, advanced prompt integration across spans and traces offers a unified connection for better prompt lineage and reproducibility, and on-demand evaluation allows for retroactive scoring of logged traces and threads. These features are complemented by opportunities for community engagement through upcoming hackathons and events aimed at connecting AI developers and enhancing skills.

LLM observability tools are essential for monitoring, tracing, and evaluating LLM applications, providing insights into their non-deterministic behavior and helping prevent unpredictable failures. These platforms enable users to trace requests, assess output quality, and monitor performance, ensuring issues are detected before affecting users. Choosing the right observability tool depends on the specific needs of the application, including whether it is in development or production, and whether it requires evaluation-centric or observability-centric features. Popular tools in 2025 include open-source options like Opik and Langfuse, which offer comprehensive observability and evaluation capabilities, and commercial platforms like Datadog and LangSmith, which provide integration with existing systems and advanced features such as real-time guardrails and automated prompt optimization. The choice between self-hosted and managed services also depends on factors like control, compliance, and cost considerations. Regardless of the platform chosen, implementing observability is crucial for ensuring reliable and performant LLM applications.

Testing a new AI assistant has revealed that while it can handle 90% of user prompts effectively, it sometimes provides incorrect answers that appear convincing but are fundamentally wrong, such as unsafe medical advice or inaccurate legal summaries. The human-in-the-loop (HITL) feedback approach is crucial in these scenarios, embedding humans at critical points in the AI lifecycle to review outputs, correct errors, and guide system behavior. This approach is essential because LLMs can make confident mistakes due to their probabilistic nature, often generating outputs that lack actual grounding. HITL helps address LLM limitations such as hallucinations, ambiguity, and edge cases, ensuring outputs are not only factually correct but aligned with domain-specific nuances, business intents, and user experiences. Human feedback is vital for maintaining ethical standards, regulatory compliance, and brand reputation, as humans can navigate complex scenarios and policy changes that models may not handle well. To operationalize HITL effectively, teams use strategies like Reinforcement Learning from Human Feedback, Active Learning, and Interactive Machine Learning, which involve structured reviews and iterative feedback loops to refine AI outputs. Tools like Opik support these processes by providing a detailed trace of AI activity and enabling cross-functional collaboration, ensuring AI systems are continuously improved and aligned with organizational goals and standards.

Context engineering is a crucial discipline for developing reliable large language model (LLM) systems, as it involves designing and optimizing the surrounding information that influences model outputs. It addresses the nearly-infinite data potential in LLM requests, optimizing the use of context windows to ensure models process relevant and accurate information efficiently. Essential components include system prompts, user input, conversation history, retrieved knowledge, tool descriptions, task metadata, examples, and hidden content. Effective context engineering requires principles like relevance, provenance, compression, hierarchical control, adaptivity, observability, efficiency, and safety. It involves nuanced decisions on managing retrieved knowledge, conversation history, and tool interactions, as well as using few-shot examples to enhance model performance. Governance and evaluation are integral to maintaining reliable, auditable, and compliant systems, using practices such as version control, access management, policy enforcement, and rigorous testing. Platforms like Opik aid in capturing and analyzing LLM interactions, providing visibility into context usage and enabling continuous improvement in context engineering.