March 2026 Summaries

Continuous profiling with tools like Pyroscope and Alloy can significantly enhance performance optimization, as demonstrated by a case study involving the TON blockchain optimization contest. This contest challenged developers to improve a blockchain validation algorithm written in C++, with performance gains measured by execution time. Pyroscope engineers used Alloy's eBPF-based CPU profiler to analyze contestant submissions, identifying performance bottlenecks without altering the code. Notable optimizations included replacing standard cryptographic libraries with faster alternatives, using hash tables instead of ordered collections, and applying algorithmic improvements to reduce computation overhead. These changes yielded performance improvements ranging from 1.5% to 20%. The case study underscores the value of modern profiling tools in making performance optimization more accessible and highlights the potential for significant speedups through relatively straightforward code modifications.

In a detailed exploration of the OpenTelemetry eBPF profiler's capabilities, the article delves into the complexities of symbolization, a crucial process for mapping memory addresses to function names in Go programs. When troubleshooting performance issues, profilers like OpenTelemetry capture raw memory addresses which are then translated into meaningful function names through a process involving the Go-specific .gopclntab section. This section, integral to the Go runtime for stack traces and reflection, remains even in stripped binaries and provides a significant advantage for profiling Go applications over other languages. The article explains how the profiler utilizes this section to efficiently resolve addresses to function names with minimal CPU overhead, thereby enabling effective continuous profiling. It highlights the challenges eBPF profilers face, such as operating in kernel space without modifying user processes, and showcases the performance optimizations like binary search and frame caching that make symbolization feasible in production environments. The discussion underscores Go's unique suitability for eBPF profiling due to its embedded symbol information, which remains intact even when binaries are stripped, facilitating full observability without compromising binary size or performance.

Grafana Labs has released version 12.4.2 along with patches for older versions, addressing critical and high-severity security vulnerabilities identified as CVE-2026-27876 and CVE-2026-27880. CVE-2026-27876, with a CVSS score of 9.1, involves the SQL expressions feature that can allow remote code execution if certain conditions are met, while CVE-2026-27880, with a CVSS score of 7.5, involves a denial-of-service vulnerability due to unbounded user input at the OpenFeature endpoint. Users are strongly advised to upgrade to the patched versions to mitigate these risks, but temporary workarounds are available. Grafana Labs coordinated with cloud providers like Amazon and Azure for early patching, and private releases were issued to customers under embargo before the public disclosure. The discoveries were made through a combination of internal and external sources, with acknowledgments given to contributors, and users are encouraged to report any further security issues through Grafana's established channels.

OpenRouter and Grafana Cloud collaborate to enhance observability in large language model (LLM)-powered applications, addressing the growing need for visibility as AI becomes integral to various systems. OpenRouter offers a unified API, allowing developers to access numerous models from providers such as OpenAI, Anthropic, Google, and Meta, while handling complexities like load balancing and model routing. The Broadcast feature automatically sends traces of API requests to observability platforms like Grafana Cloud, eliminating the need for additional instrumentation in code. These traces include detailed information such as model data, token usage, timing, cost, and errors, enabling teams to monitor performance, track costs, and debug issues effectively. Grafana Cloud users can leverage these insights through dashboards and alerts, enhancing decision-making regarding model usage, latency, and capacity planning. Additionally, the integration supports privacy controls and custom metadata, allowing teams to tailor their observability to specific application contexts.

Monitoring large language models (LLMs) in production involves distinct challenges compared to demo versions, such as managing costs, maintaining latency within service-level objectives, and ensuring the system's safety from issues like hallucinations and prompt-injection attacks. Grafana Cloud, combined with OpenLIT and OpenTelemetry, offers a comprehensive AI observability solution that visualizes and queries metrics, logs, and traces tailored to AI workloads. This setup supports monitoring model latency, throughput, cost management, and safety evaluations like toxicity and bias detection. OpenLIT facilitates easy instrumentation of AI applications, supporting a wide range of generative AI tools, and allows seamless integration with Grafana Cloud's dashboards for complete visibility over AI stack performance, including vector database operations, MCP servers, and GPU performance. The guide further demonstrates configuring Grafana Cloud to monitor a customer support chatbot, showcasing how this integration can optimize costs and reduce latency, while providing actionable insights into performance and quality issues.

In a rapidly evolving world driven by AI, observability practitioners must adapt to monitor AI applications effectively, particularly AI agents. This text outlines how Grafana Cloud, in conjunction with the OpenLIT SDK, provides comprehensive monitoring solutions for AI agents by capturing agent-level telemetry and visualizing it via prebuilt dashboards. These tools offer insights into agent actions, tool calls, reasoning paths, and associated costs, enhancing performance optimization, troubleshooting, and quality assurance. Grafana Cloud integrates with OpenTelemetry standards to deliver distributed traces and metrics, enabling users to track response times, error rates, and token usage while ensuring compatibility with evolving conventions. The guide further explains how to set up and use these tools to collect and analyze data, emphasizing the importance of full sequence visibility, cost tracking, and performance optimization in managing AI agents.

Monitoring Model Context Protocol (MCP) servers using OpenLIT and Grafana Cloud is essential for maintaining effective AI applications, as MCP servers facilitate communication between large language models (LLMs) and external tools but add complexity. The guide outlines how to use Grafana Cloud for observability of MCP servers, which are crucial in understanding latency issues, silent failures, and cross-service visibility due to their role in routing tool calls to multiple services. By leveraging OpenTelemetry and OpenLIT for instrumentation, Grafana Cloud provides pre-built dashboards that offer insights into tool performance, protocol health, resource usage, and error tracking. This enables users to trace the entire pathway of requests, manage scalability, control costs, and ensure security and compliance, ultimately improving the reliability and efficiency of AI systems. The process involves setting up OpenLIT for auto-instrumentation, configuring environment variables, and exploring the MCP observability dashboards to optimize server resources and address potential issues.

As AI technology evolves, developers increasingly use Large Language Models (LLMs) to create AI-powered applications, necessitating effective observability tools to monitor these complex systems. The OpenLIT Operator facilitates zero-code observability for AI workloads on Kubernetes by automatically integrating OpenTelemetry instrumentation, eliminating the need for manual code changes. This approach, when combined with Grafana Cloud, allows for comprehensive monitoring of AI applications, covering aspects such as latency, cost, token usage, and agent workflows. By leveraging OpenTelemetry standards, the OpenLIT Operator supports various AI frameworks and providers, ensuring seamless integration with existing observability infrastructures and enabling vendor-neutral telemetry management. Grafana Cloud further enhances this capability by offering pre-built dashboards and alerting systems to visualize and manage performance metrics, providing a streamlined solution for maintaining and optimizing AI services without altering the application code.

A recent blog post by Lea Nagashima, published on March 19, 2026, briefly highlights upcoming content related to monitoring large language models (LLMs) in production using Grafana Cloud, OpenLIT, and OpenTelemetry. It also mentions the monitoring of Model Context Protocol (MCP) servers with these tools and introduces the concept of zero-code observability for LLMs and agents on Kubernetes. The blog connects readers with related content, including videos and documentation on performance testing, logging, and frontend observability with Grafana Cloud, as well as product insights for Grafana Cloud AI/ML and Grafana k6.

Open standards and open source software have become integral to modern observability strategies, as demonstrated by Grafana Labs' 2026 Observability Survey. The survey highlights a significant shift from vendor-specific solutions to open, portable standards, with 77% of respondents valuing open source and open standards in their observability strategies. Key technologies like Prometheus and OpenTelemetry are central to this trend, with organizations increasingly investing in these tools to unify telemetry data across systems. OpenTelemetry, in particular, is gaining momentum for its broad application across metrics, logs, and traces, despite being less mature than Prometheus. The survey also reveals a growing interest in the intersection of open source and AI, as well as the desire for observability solutions that offer simplicity and flexibility, enabling easy adoption and vendor switching. This evolution is driven by the need for better data correlation and management across diverse platforms, underscoring the expanding role of observability in business operations.

AI's role in observability is expanding rapidly, as highlighted by the 2026 Observability Survey conducted by Grafana Labs, which underscores both significant potential and notable reservations. Practitioners see AI's value in anomaly detection, trend forecasting, root cause analysis, and onboarding, yet express skepticism about granting AI autonomous decision-making capabilities, especially in smaller companies. Despite the enthusiasm for AI's potential productivity boost, the primary hurdle remains the friction from excessive manual input required for AI tasks. Trust plays a crucial role, with 95% of respondents emphasizing the importance of AI transparency and reasoning to ensure accountability. While AI capabilities are not yet a top criterion for selecting observability tools, cost and ease of use currently take precedence. Adoption of large language model (LLM)-based applications is increasing but still nascent, with only 14% using them for production workloads, reflecting a varied pace of AI integration across the industry.

Modern runtime security is essential for understanding complex systems and protecting against attacks in cloud-native architectures, but achieving deep visibility into runtime risks is challenging. Grafana Labs and Miggo Security have partnered to deliver a joint solution that provides evidence-based runtime security using existing production telemetry in Grafana Cloud, without additional runtime costs or operational overhead. Traditional security methods often result in a low true positive rate of exploitable vulnerabilities and overload security teams with irrelevant alerts, eroding trust with developers. The industry is shifting towards runtime security, which focuses on actual running services to identify vulnerable code, but this approach faces challenges such as deployment tax, lack of runtime context, and incomplete vulnerability records. Grafana Cloud and Miggo's integration leverages existing telemetry data to address these challenges by collecting, analyzing, prioritizing, and mitigating vulnerabilities, offering a more efficient and trustworthy security solution. The partnership aims to improve the security landscape by embedding security into observability workflows, with plans to showcase their solution at RSAC 2026.

Grafana has introduced new one-click integrations in its Drilldown apps, enhancing the user experience by seamlessly connecting exploration and action within its observability platform. These updates aim to eliminate the need for manual query copying and context switching by integrating Drilldown directly with core Grafana features, thus reducing the time from insight to action and improving cross-team collaboration. New features include the ability to add panels directly to dashboards, create alerts from metric panels with ease, and save searches and queries for future use, all of which streamline workflows and lower the barrier for teams new to using PromQL, LogQL, or TraceQL. Additionally, the enhanced display for OpenTelemetry log lines in Logs Drilldown offers a more organized view of important data, making logs easier to scan and understand. These enhancements, available in the latest versions of Grafana and its Drilldown apps, are designed to provide a more efficient and cohesive experience for users, whether they are seasoned site reliability engineers or newcomers.

In a cloud-native environment, managing resources and optimizing costs can be streamlined using Grafana Assistant integrated with Model Context Protocol (MCP) servers. Grafana Assistant, an AI-powered agent within Grafana Cloud, allows users to automate cost optimization processes by analyzing telemetry data without complex queries, producing an analysis of resource utilization, and offering actionable recommendations. It identifies resource waste and optimization opportunities, which can be automatically implemented via MCP server integrations with systems like GitHub, enabling direct updates to codebases. This system supports various cloud platforms like AWS, GCP, or Azure and can extend beyond Kubernetes workloads to any telemetry-generating resource. The open nature of MCP allows further integration with external systems for varied insights, making Grafana Cloud a versatile tool for managing metrics, logs, and dashboards.

Alloy has integrated an experimental OpenTelemetry engine in its v1.14.0 release, allowing users to configure and run Alloy using standard OpenTelemetry Collector YAML while maintaining backward compatibility with existing setups. This move enables Alloy to offer both its native capabilities and the unified OpenTelemetry standards, enhancing flexibility for users who wish to leverage open standards alongside Alloy's operational strengths. The new engine is opt-in and can be tested without altering current Alloy configurations, allowing parallel operation of Alloy's native and OpenTelemetry pipelines within a single instance. This update aims to provide a seamless experience for users familiar with OpenTelemetry while retaining the performance optimizations and integrations Alloy offers, especially with Grafana's observability stack. The developers plan to refine operational parity between the two engines based on user feedback to ensure a smooth migration and consistent user experience.

Apono's integration with Grafana aims to enhance security by providing Just-in-Time (JIT) access to data sources, ensuring that users receive time-bound and least-privilege access only when necessary. This approach addresses the challenges posed by static, always-on access models, which can lead to over-privileged access and compliance issues in dynamic environments. Apono integrates with various data sources connected to Grafana, such as logs, SQL databases, and cloud metrics, allowing security and platform teams to define policies determining access eligibility, duration, and conditions. This system temporarily provisions permissions and automatically revokes them, maintaining operational workflows while reducing security risks. The integration is beneficial for organizations seeking to improve both security and operational efficiency, as it limits the exposure of sensitive data, supports compliance through full audit visibility, and enhances the speed of investigations by streamlining access requests. Available for both on-premises and Grafana Cloud, this integration helps manage access dynamically, aligning with real-time operational responsibilities and eliminating standing privileges.

The 2026 Golden Grot Awards highlight innovative dashboards created with Grafana, showcasing the remarkable applications of observability across diverse domains. Finalists include projects that transform raw data into actionable insights, such as a homelab observability playground, a planetary health monitoring system, a cardiac recovery tracker, a marathon training diagnostic tool, and an aurora borealis visibility predictor. In the professional category, standout entries include a scalable firewall analytics platform, a global arcade game analytics dashboard, lunar mission telemetry monitoring, a real-time automotive diagnostics tool, a seismic network monitoring ecosystem, and a government service performance tracker. Public voting is open until March 11, with winners to be celebrated at GrafanaCON 2026 in Barcelona.

GrafanaCON 2026, scheduled for April 20-22 in Barcelona, will feature a diverse agenda encompassing major announcements, technical deep dives, and community success stories, showcasing the versatility of Grafana and its open-source ecosystem. The event will include hands-on labs, Ask the Experts booths, and a variety of sessions focusing on Grafana, Prometheus, OpenTelemetry, Loki, and more, alongside the annual Golden Grot Awards. Key highlights include the unveiling of Grafana 13, sessions on AI agent context, and innovations like Loki's new architecture and Alloy's OpenTelemetry Engine. The conference will also spotlight community-driven projects, such as ocean monitoring with DIGI4ECO and the LEGO Group's dashboard framework. Additionally, attendees can engage in networking activities like the Community Garden Party and the GrafanaCON Science Fair, making it an essential gathering for those involved with Grafana and open observability.

Tempo, an open-source distributed tracing backend, has introduced a powerful feature in its 2.10 release that addresses the challenge of generating metrics directly from traces, specifically focusing on managing cardinality. Through the optional metrics-generator component, Tempo can produce RED metrics from traces and push them to Mimir, filling the gap for applications that use tracing without metrics. However, this capability risks causing a cardinality explosion—an overwhelming number of unique metric series that can increase costs. To tackle this, Tempo employs HyperLogLog, a probabilistic data structure that efficiently estimates cardinality with minimal memory usage. This approach allows for significant memory savings, providing estimates with only a 3% error margin. Tempo's integration with Grafana Cloud further enhances cardinality management by enabling users to visualize, manage, and forecast metric usage, aiding in decision-making around quota adjustments and cost control.

Grafana Cloud Synthetic Monitoring now offers a secrets management feature that enhances security by allowing users to securely store and manage sensitive credentials like API keys, passwords, and tokens for synthetic monitoring checks. This feature encrypts sensitive data at rest and automatically redacts it in logs and outputs, while providing centralized storage for easy credential management and rotation. Grafana Cloud users with the appropriate roles can create, edit, and use secrets in their synthetic monitoring tests, ensuring that sensitive data is protected during API testing, user workflow validation, and external service access. The feature supports various check types, with future plans to expand its applicability, and offers a straightforward setup process that includes importing and referencing secrets in scripts, organizing checks with labels, and selecting probe locations. Automatic redaction of secret values in logs and outputs prevents accidental exposure, and users can manage their secrets through a dedicated interface, updating or deleting them as needed without disrupting existing tests. This addition to Grafana Cloud is available to all users, offering a secure and scalable way to handle authentication credentials in monitoring infrastructure.

Grafana Alerting has undergone significant updates to enhance its usability and efficiency, particularly for managing large volumes of alert rules. The revamped UI now includes a faster, paginated alert rules page with a consolidated filter dropdown and saved searches for personalized views, transforming it from a mere configuration tool into a comprehensive operational workspace. The introduction of alert rules v2 provides a cleaner interface to handle thousands of rules efficiently, while the refined filtering UX reduces visual clutter and maximizes screen space. Saved searches simplify repetitive filtering tasks by allowing users to store and easily access their preferred filter settings. Additionally, the new alert activity page offers flexible grouping and dedicated detail drawers, enabling operators to focus on system status and incident resolution without being overwhelmed by configuration details. These enhancements collectively aim to streamline workflows and improve the on-call experience for users.