August 2025 Summaries

Project Bob is an ambitious endeavor led by Andrew McCalip, aiming to be the first autonomous boat to circumnavigate the globe, utilizing open-source tools and a custom Grafana Cloud dashboard for monitoring. The 14-foot solar-powered vessel will rely on real-time data collection through onboard cameras, sensors, and a Starlink antenna to stream live video and telemetry data to the public. McCalip, known for his innovative projects and extensive online following, initiated Project Bob through a successful Kickstarter campaign, leading a team of engineers to design an energy-efficient, self-sustaining drone boat capable of recovering from failures autonomously. The boat operates on a Raspberry Pi 4 with multiple Docker containers and uses a robust data pipeline to ensure its functionality and public engagement. Despite challenges such as Wi-Fi disconnections and the necessity for the boat to travel over 25,000 miles across major oceans without human intervention, the team is determined to achieve this world-record-setting goal.

Victoria Nduka's research project, conducted during her mentorship with Prometheus through the Linux Foundation, explored how Prometheus handles OpenTelemetry resource attributes and sought user feedback to improve the experience. Her study revealed that while Prometheus uses labels to identify metrics, converting OpenTelemetry resource attributes to labels can lead to cardinality issues. To address this, three main approaches are considered: mapping all resource attributes to labels, selective promotion of important attributes, and using a separate metric called "target_info" for joining queries. Stakeholder and user interviews highlighted the lack of early collaboration between the Prometheus and OpenTelemetry communities, leading to interoperability challenges. Users found the current integration complex, particularly the join syntax, and expressed that documentation gaps hinder their understanding. The study emphasized the need for improved documentation, awareness of OpenTelemetry's resource detection feature, and potential long-term solutions like metadata storage and an entity framework. These findings aim to enhance user experience without compromising Prometheus's core architecture, with a focus on fostering better collaboration between the two communities.

HTTP Performance Insights is a new feature in Grafana Cloud Frontend Observability designed to enhance application performance by analyzing HTTP requests. It provides detailed insights into request quality and bottlenecks, allowing frontend engineers to proactively identify and resolve performance issues before they impact user experience. The tool offers a rich interface with health indicators, trend analysis, and sortable tables to pinpoint slow endpoints and error-prone calls, facilitating efficient troubleshooting. By enabling request-level granularity, it aids in precise analysis of problematic endpoints, helping correlate performance with user sessions or geographic regions. Available in public preview, the feature requires no additional instrumentation and integrates seamlessly with existing HTTP data, representing the initial phase of a broader performance intelligence platform. Future enhancements aim to include page performance insights and resource loading intelligence for a comprehensive performance analysis suite.

Grafana Labs' Voice of Customer (VoC) program is a structured initiative aimed at gathering and acting on product feedback from its users to ensure the company's offerings align with diverse customer needs. Originally a simple spreadsheet for logging requests, the VoC program has evolved into a comprehensive system involving dedicated teams that analyze feedback and inform product development. Notably, about 35% of the feedback received has led to new features or initiatives, such as expanded role-based access control, improved cost visibility, and enhanced programmatic workflows. Recent enhancements include a customer-facing Feature Request Portal that allows users to submit, view, and track feedback more transparently, fostering better community engagement. The program also plans to integrate open-source community feedback more systematically to capture a wider range of user insights. Through these efforts, Grafana Labs aims to build products that reflect the needs of both its cloud and open-source user communities, ensuring that customer use cases are met with confidence and precision.

Canary tokens, used by Grafana Labs during a security incident involving a GitHub Action workflow, serve as digital decoys that alert security teams to potential intrusions by mimicking valuable assets. These tokens, which can appear as API keys, files, URLs, or DNS entries, triggered an alert when an attacker used an AWS API key, allowing for rapid containment of the breach without any compromise to production systems or customer data. The strategic placement and integration of canary tokens within Grafana Labs' infrastructure, using Thinkst's platform, enabled real-time alerts through Slack and facilitated an effective response by the Detection & Response team. The blog emphasizes the importance of careful token placement, integration with automated response workflows, and the use of metadata for efficient triage, while also addressing potential limitations such as false positives and infrastructure vulnerabilities. Additionally, Grafana Labs highlights best practices for implementing a canary token strategy, including mixing token types, automating placement, and educating teams to enhance overall security posture.

Goutham Veeramachaneni from Grafana Labs developed a homelab to address observability challenges and discovered the complexities of self-hosted services, which often lack consistent metrics for monitoring. To tackle these issues, he utilized Grafana Beyla, an open-source eBPF-based tool, which provides zero-code instrumentation for Linux applications, enabling better observability. Beyla, in combination with eBPF and OpenTelemetry, simplifies the monitoring process by collecting telemetry data from applications at the kernel level, overcoming limitations of traditional instrumentation. It allows for the extraction of telemetry without requiring code changes, which is especially useful for uninstrumented or legacy applications. While not as context-rich as native instrumentation, Beyla offers a fast start towards comprehensive monitoring, making observability more accessible for both homelabs and large-scale production environments. Beyla's integration with OpenTelemetry further supports its role in providing consistent and powerful insights, positioning it as a valuable tool for both enthusiasts and professionals in the field of observability.

Grafana Mimir, a horizontally scalable and multi-tenant time-series database for Prometheus metrics, can be deployed on Kubernetes or bare metal, with the latter offering certain advantages in specific scenarios. Running Mimir on bare metal or virtual machines is beneficial when a Kubernetes cluster is unavailable, as it allows for deployment using tools like Puppet or Ansible, and it helps avoid circular dependencies that could arise when providing internal metrics services to Kubernetes teams. Additionally, deploying on bare metal provides an opportunity to delve deeper into software functionalities without relying on Kubernetes' extensive features and automation. Although deploying on bare metal might require more manual configuration and increase the complexity of maintenance, it can be a viable option depending on organizational needs and expertise. The choice between Kubernetes and bare metal deployment should be guided by a thorough assessment of the benefits and potential drawbacks, supported by proof-of-concept testing to identify the most suitable approach.

Grafana Labs has integrated Salesforce as an Enterprise data source, allowing users to visualize Salesforce data within Grafana dashboards and correlate it with other data sources. This integration supports three methods for querying Salesforce data: a SOQL editor for advanced users, a query builder for those preferring a point-and-click interface, and a reports mode for quick access to Salesforce reports. Additionally, the Salesforce Enterprise data source features annotations, templates, variables, transformations, alerting, and supports both credentials-based and JWT-based authentication. Built and maintained by Grafana Labs, these Enterprise data sources are available across all Grafana Cloud tiers, including the free tier, enabling users to gain insights by connecting various external systems and proprietary platforms.

Grafana Cloud has introduced several updates to enhance its fully managed observability platform, powered by the open-source Grafana LGTM Stack. Notable new features include the AI-powered Grafana Assistant, which aids in query and dashboard creation, and a secrets management system for secure credential storage in Synthetic Monitoring. The platform now supports the Model Context Protocol (MCP) in Grafana Cloud Traces, allowing for LLM-powered insights into tracing data. It also integrates with Tailscale for secure, private data source queries and offers out-of-the-box alerting for Frontend Observability. Enhanced incident management features, including status updates and a unified experience for incident webhooks, are now generally available, while k6 extensions for performance testing and new integrations with Anthropic and Jenkins offer expanded monitoring capabilities. These developments aim to simplify onboarding, improve productivity, and provide comprehensive insights into system operations, with a generous free tier available for new users.

Grafana Cloud has introduced a new secrets management feature to address the challenge of securely storing credentials like API keys, passwords, and tokens in observability stacks. Currently in public preview, this feature aims to centralize credential storage with enterprise-grade encryption and offer benefits such as reference-based secrets access, granular access controls, and a complete audit trail of secret usage. Initially integrated with Grafana Cloud's Synthetic Monitoring, this system replaces the insecure practice of embedding credentials in code with a more secure method of referencing secrets by name. Grafana Cloud's secrets management is designed with robust security measures, including envelope encryption, metadata separation, and strict access controls. The feature not only provides a secure way to manage secrets but also lays the groundwork for future integrations across the Grafana ecosystem. This development is part of Grafana Labs' commitment to enhancing security infrastructure and expanding capabilities to meet the evolving needs of their users.

The Anthropic integration for Grafana Cloud is a new tool designed to monitor and manage usage and costs related to Claude large language models (LLMs) by connecting directly to the Anthropic Usage and Cost API. This integration enables users to gain real-time insights into model performance and expenses through a pre-built dashboard and customizable alerts within Grafana Cloud. It offers precise billing insights, token usage analysis, model performance evaluation, and capacity planning to optimize resource allocation and manage costs efficiently. The integration is collector-less, leveraging Grafana Cloud's Metrics Endpoint to convert usage data into Prometheus-format metrics without requiring additional agents. Key features include a pre-built API usage dashboard that facilitates easy tracking of token consumption and costs and customizable alert rules to monitor unexpected usage spikes. Users can set up the integration by configuring a scrape job with their Anthropic Admin API key, allowing Grafana Cloud to automatically collect usage metrics. Beyond the pre-built functionalities, users can create custom monitoring panels and optimization strategies to enhance usage efficiency and budget control across different projects or teams.

Andy Jiang discusses the integration of Deno and Grafana Cloud to enhance observability in JavaScript projects without the need for manual code instrumentation. Deno, a modern JavaScript and TypeScript toolchain, offers built-in OpenTelemetry support that automatically collects metrics, logs, and traces, simplifying the process of monitoring applications. By connecting Deno with Grafana Cloud, developers can set up a system that routes telemetry data through an OpenTelemetry collector, enabling them to filter logs, set up tracing for distributed systems, and obtain built-in metrics with minimal effort. This integration allows for seamless tracking of requests across multiple services, automatic propagation of trace information, and the collection of valuable metrics, providing a robust foundation for understanding and improving the performance and reliability of JavaScript projects.

Grafana Assistant is a newly launched AI-powered tool available in public preview for Grafana Cloud users, designed to streamline the onboarding process for teams and enhance their experience with the Grafana platform. The assistant acts as an integrated chat agent that aids users by answering questions, providing navigation help, and executing tasks within the Grafana environment, thereby reducing the learning curve and increasing adoption speed. It offers context-aware assistance tailored to the user's specific observability stack and is capable of comparing concepts across different platforms, offering customized tutorials, and sharing best practices. Grafana Assistant is part of Grafana Labs' broader AI strategy, which emphasizes human-in-the-loop interactions and aims to enhance workflows by simplifying common tasks such as query writing, dashboard creation, and anomaly investigation. While not open-source, it is available for Grafana Cloud, with the company continuing to invest in open-source tools to allow community-driven AI-powered experiences. The assistant is expected to evolve with user feedback and ongoing development, aiming to provide more autonomous capabilities as the technology matures.

Grafana Labs is integrating AI into observability to enhance the accessibility and effectiveness of monitoring systems for a diverse range of users, from operators to developers. By leveraging AI, particularly in tasks involving large volumes of data, Grafana aims to streamline workflows such as incident management, routine check-ups, and dashboard creation. The company emphasizes the importance of human-in-the-loop (HITL) systems, ensuring AI serves as an assistive tool rather than a replacement for human judgment. Grafana's open-source nature provides it with an advantage as AI models can be trained on publicly available resources, enabling rapid model development and improved performance. The introduction of Grafana Assistant exemplifies this approach, offering users AI-driven support for building queries and dashboards while maintaining control. Grafana Labs is committed to evolving AI capabilities, focusing on outcome-driven experiences that redefine traditional roles and tasks, and positioning AI as a long-term investment to democratize observability practices.

Grafana Cloud Traces, a fully managed distributed tracing system powered by Grafana Tempo, now supports the Model Context Protocol (MCP), a protocol developed by Anthropic to standardize context provision for large language models (LLMs). This integration enables users to leverage LLM-powered tools like Claude Code and Cursor to analyze and derive insights from tracing data, simplifying the process of understanding service interactions and diagnosing issues. The post provides a walkthrough on connecting Claude Code to Grafana Cloud Traces, utilizing the MCP endpoint, and demonstrates how tracing data can be used for onboarding, identifying errors, and optimizing service performance. Additionally, the integration is in public preview and will be included in the upcoming Grafana Tempo 2.9 release, with future plans to enhance data presentation and API capabilities for more efficient LLM interactions.

Brazilian organizations are increasingly adopting observability trends and technologies to enhance their software systems, as revealed by a localized survey conducted by Grafana Labs. This marks the first time the survey has been tailored specifically for Brazil, providing insights into the Brazilian market's use of multiple observability tools and data sources, with 96% of organizations using multiple technologies and a significant portion managing their own observability stacks. The survey highlights the role of open standards and open-source software in facilitating data integration, with Prometheus and OpenTelemetry being widely used. While the Brazilian market exhibits varying degrees of observability maturity, there is a noticeable inclination toward emerging practices such as Service Level Objectives (SLOs) and FinOps, with a focus on reducing mean time to resolution and managing costs. Complexity remains a major concern, and Brazilian organizations prioritize cost and ease of use when selecting observability tools, with a growing interest in AI/ML functionalities. The survey, conducted at ObservabilityCON in São Paulo, gathered responses from a diverse range of industries and roles, reflecting a broad interest in improving observability practices.

Game of Traces is an innovative grand strategy game designed to teach engineers the fundamentals of OpenTelemetry tracing by simulating interactions within a microservice architecture. Developed by Jay Clifford and Tom Glenn at Grafana Labs, this game uses Python Flask servers to represent capitals and villages, allowing players to collect resources, create armies, and engage in strategic battles against an AI opponent. Each in-game action is tracked as a trace, composed of spans that provide insights into service calls and interactions. The game leverages the Grafana LGTM Stack to visualize and correlate telemetry signals, such as traces, logs, and metrics, enabling players to understand complex observability concepts. Game of Traces also introduces the use of span attributes and links to illustrate the flow of events and relationships between spans, offering a comprehensive learning experience for those new to observability.

Wilfried Roset, an engineering manager at OVHcloud and a Grafana Champion, illustrates how SQL can be used to extract and analyze data from Grafana's database, which is a repository of information about users, dashboards, and data sources. Grafana relies on a database to store its objects, and SQL queries can serve as an internal API to access the same data that Grafana's public HTTP API can expose. By using SQL, users can gather insights into Grafana's usage patterns, such as the number of organizations, users, and dashboards, and identify the most utilized data sources and panel types across dashboards. Roset emphasizes that while SQL provides powerful capabilities for understanding user behavior and system usage, modifications to the database should be avoided to prevent potential disruptions. He highlights that the exploration of Grafana's database with SQL can reveal valuable information, although complex queries might be required for deeper analysis.

Grafana Cloud has introduced a feature called deletion protection to prevent the accidental removal of critical resources within its observability stack. This safeguard is designed to block deletion attempts across multiple channels, including the Grafana Cloud UI, Terraform, Crossplane, and Ansible providers, requiring manual disabling before any deletion can occur. Starting from recent versions, all newly created Grafana Cloud stacks have this protection enabled by default, ensuring that users must explicitly disable it before stack deletions. The feature is integrated into Grafana's infrastructure as code offerings, allowing users to manage deletion protection through declarative configurations in Terraform, Crossplane, and Ansible. This default protection also applies when upgrading to newer versions of these providers, ensuring backward compatibility and promoting robust management of observability resources. Users are encouraged to apply changes to maintain deletion protection, although they can disable it if desired. For further information, Grafana Cloud provides technical documentation, emphasizing its role as an accessible platform for managing metrics, logs, and dashboards.

ObservabilityCON 2025, Grafana Labs' flagship observability event, will take place in London from October 7-9, but for those unable to attend, a one-day version called ObservabilityCON on the Road will be held in various cities, including São Paulo, Washington, D.C., Dallas, and Madrid, offering a chance to explore new AI features and Grafana Cloud solutions. The event promises technical deep dives, live demos, and an "Ask the Experts" booth, providing opportunities to enhance observability strategies and connect with industry leaders. Participants can benefit from early bird registration discounts, and the event aims to deliver insights into application performance and foster networking through hands-on demonstrations and regional success stories.

Grafana has introduced a redesigned alert rules list page aimed at improving the management of alerts by offering a faster, more intuitive, and scalable experience for users. The update features two distinct views: a grouped view for understanding the structure of alerting setups and a list view for quick searching and filtering of alert rules. This redesign focuses on enhancing performance by utilizing a new paginated API, which ensures a responsive experience regardless of the number of rules. The interface has been streamlined to emphasize core functionalities, and a layered data approach combines fast access with consistent updates. Additionally, the redesign addresses challenges related to stable identifiers and frontend filtering, providing a more effective way to manage alerts. Future developments will focus on improving the troubleshooting experience for active alerts, aiming to provide users with better context for resolving issues quickly.

Grafana Beyla 2.5, an open-source eBPF-based zero-code instrumentation tool, marks its evolution by integrating closely with the OpenTelemetry eBPF Instrumentation project, which Grafana Labs has donated. This release features MongoDB protocol support, enabling automatic capture of database operations across various programming languages without code changes, and JSON-RPC support for Go applications, enhancing visibility into service interactions within the blockchain ecosystem. Additionally, Beyla 2.5 introduces manual span support for Go applications, allowing for custom observability data, and enhances NodeJS distributed tracing, ensuring better trace correlation across mixed technology stacks. A new survey mode aids in service discovery, helping organizations understand their application landscape before full instrumentation. This version includes bug fixes and performance improvements, reinforcing Grafana Labs' commitment to maintaining compatibility with OpenTelemetry standards and fostering community contributions.

On July 31, 2025, Grafana Labs addressed a memory corruption issue in Grafana Cloud Metrics caused by a bug in the Mimir distributor update, which led to a potential data leak involving error messages. The bug, introduced in a recent release candidate, allowed fragments of metrics data from one customer to appear in error messages of another customer, particularly affecting those using the Insights UI. The issue was promptly patched, and erroneous messages were scrubbed from the system by August 3, 2025, with impacted customers notified. Grafana Labs has taken steps to prevent such incidents by improving error handling and security measures, including a review of gRPC buffer pooling and enhancement of static application security testing (SAST) tools. They continue to focus on transparency and security by analyzing any compromised data and reinforcing security protocols. The incident did not affect OpenTelemetry Protocol users, and Grafana Labs has urged users to rotate sensitive information stored in metrics gathering scripts.