April 2025 Summaries

The Ingress NGINX Controller is a key Kubernetes component for routing HTTP and HTTPS traffic, offering extensive observability through Prometheus metrics, but it often results in an overwhelming number of time series due to its detailed labeling of metrics by attributes like ingress name, host, and path. This high cardinality can become problematic in high-traffic or large-scale environments, necessitating strategies to manage metric volumes without losing vital observability signals. Key tactics include disabling certain labels, aggregating metrics, and focusing on essential metrics, which can significantly reduce the number of time series while maintaining the necessary performance and traffic insights. Grafana Cloud provides tools such as cardinality management dashboards and Adaptive Metrics to further control metric growth and costs, emphasizing the importance of balancing observability with resource efficiency. By implementing these strategies, users can ensure effective monitoring of their Ingress NGINX Controller even as their Kubernetes clusters expand.

Grafana Labs recently detected a security incident involving a GitHub workflow vulnerability that allowed unauthorized access to a limited number of tokens, which were subsequently invalidated. The breach was identified when a canary token was triggered, prompting an immediate investigation by the security team. The attack involved forking a repository to inject malicious code and extract tokens, but no customer data or production systems were compromised. Grafana Labs responded by removing the affected GitHub Action, disabling public repository workflows, rotating exposed tokens, and auditing their internal workflows for similar vulnerabilities. The company is also enhancing its CI/CD security measures and has launched a full audit of access logs to ensure containment. They emphasize the importance of reporting security issues through their dedicated page and request non-disclosure of vulnerabilities until a fix is announced. Security updates and remediations are published on their blog.

The recently introduced Causely data source plugin for Grafana aims to enhance observability by providing real-time analysis of cause-and-effect relationships within applications, allowing teams to understand the underlying reasons for application degradation rather than merely identifying symptoms. The plugin integrates seamlessly into Grafana dashboards, which are widely used by engineers for consistent visualization and alerting, enabling users to view root causes alongside metrics, traces, and logs without switching tools. Causely's commitment to open source is evident through its contributions to the OpenTelemetry community and the development of Beyla, an eBPF-based auto-instrumentation agent that collects traces and metrics from Kubernetes environments efficiently. This integration with Grafana not only enriches alerts with diagnostic context via Alertmanager but also allows teams to quickly identify and address issues, ensuring service reliability and reducing manual troubleshooting. By building on open standards and fostering a community-driven approach, Causely strives to make observability tools more insightful and actionable for developers.

Booking.com, a leading online travel platform, underwent a significant transformation in its observability strategy by consolidating multiple observability solutions into a unified platform with Grafana Cloud. This transition, driven by the need for a scalable, cost-efficient, and modernized system, enabled Booking.com to centralize its telemetry data, including metrics, logs, traces, and profiles, using OpenTelemetry and as-code practices to enhance automation and consistency. The move also involved replacing various homegrown and costly solutions with vendor-agnostic pipelines that could support both on-premises and cloud-based services. This shift aimed to improve real user monitoring and profiling capabilities, thereby enhancing application performance and customer satisfaction. Throughout this process, Booking.com emphasized the importance of teamwork, transparent communication, cost analysis, and continuous education to ensure a successful adoption of the new observability strategy.

GrafanaCON 2025, taking place from May 6-8 in Seattle, promises to be the largest gathering yet for the Grafana community, featuring a diverse array of sessions, workshops, and networking opportunities centered around the Grafana open-source ecosystem. Highlights include Firefly Aerospace's presentation on monitoring spacecraft systems during a moon mission, as well as contributions from EA Apps, Dropbox, and other organizations showcasing their Grafana use cases. Attendees can engage in hands-on labs, technical deep dives, and informal "Birds of a Feather" discussions, while the event also offers unique experiences like a Science Fair, a party at the Museum of Pop Culture, and the Golden Grot Awards celebrating top dashboards. With a focus on learning, collaboration, and community building, GrafanaCON 2025 aims to create an inclusive and dynamic environment for all participants, complete with quiet spaces, food, and coffee stations to enhance the experience.

Frontend observability is vital for gaining insights into real user interactions with web applications, enabling developers to identify and resolve issues that might not surface during development and testing. The blog post introduces Grafana Faro, an open-source web SDK designed to enhance frontend observability by collecting and transmitting performance data such as web vitals, logs, and user events to Grafana Cloud. This tool is lightweight, integrates easily with popular JavaScript frameworks, and offers flexible deployment options, either through an open-source approach with Grafana Alloy or a managed solution via Grafana Cloud, which provides ready-to-use dashboards for monitoring. The post provides a step-by-step tutorial on using Grafana Faro with a demo app, emphasizing its capacity to improve website health and user experience by offering developers real-time visibility into key performance metrics.

Grafana Labs has released a series of security patches for various versions of Grafana, addressing both high and medium severity vulnerabilities: CVE-2025-3260, CVE-2025-2703, and CVE-2025-3454. CVE-2025-3260, with a CVSS score of 8.3, allows users with Viewer and Editor roles to bypass dashboard-specific permissions, impacting Grafana 11.6.x and requiring an upgrade to mitigate. CVE-2025-2703, scoring 6.8, involves a DOM XSS vulnerability in Grafana’s XY chart plugin, affecting versions from 11.1.0 and mitigated by upgrading or enabling Trusted Types. CVE-2025-3454, with a CVSS score of 5.0, affects the data source proxy API, allowing unauthorized read access to certain endpoints, impacting versions from 8.0 and requiring an upgrade or a reverse proxy as a solution. Grafana Cloud instances have already been patched, and both Amazon Managed Grafana and Azure Managed Grafana confirmed their security. Users are urged to update to the appropriate patched versions and can find detailed incident timelines and mitigation steps on Grafana’s blog.

Adaptive Logs in Grafana Cloud has introduced new features to help users optimize log volumes and costs more effectively while maintaining flexibility and control. These enhancements include the ability to temporarily pause Adaptive Logs to retain full log access during critical incidents or updates, as well as permanent exemptions for certain logs necessary for compliance or audit purposes. Users can now apply per-service drop rates to manage log ingestion with greater precision, allowing organizations to balance optimization with the specific needs of different services and teams. These features are designed to give observability teams confidence in managing their logs without sacrificing important data, and they are available in public preview for all Grafana Cloud customers.

Marie Cruz, a Senior Developer Advocate at Grafana Labs, introduces her daughter to IoT, monitoring, and observability through a gardening project. Together, they set up an IoT system to monitor plant moisture levels using a soil moisture sensor, Arduino, Prometheus, and Grafana Cloud. The project involves connecting the sensor to an ESP32 development board and using Arduino IDE to write a program that reads sensor data and determines soil moisture status. Prometheus collects and stores the data, while Grafana Cloud visualizes it through dashboards and alerts. This interactive project not only enhances her daughter's understanding of IoT and tech concepts but also showcases how technology can be applied in everyday activities such as gardening.

Cookidoo, an online platform by Vorwerk featuring over 80,000 guided recipes for the Thermomix, upgraded its observability plan by transitioning from Instana to Grafana Cloud and OpenTelemetry to address its monitoring needs. This shift was driven by the challenges faced in integrating Instana with other tools and the inefficiencies in incident response. By moving to Grafana Cloud, the team achieved cost savings, improved incident response, and automated dashboards, while also incorporating business metrics into their observability strategy. The transition involved standardizing dashboards and metrics, promoting the use of traces, and leveraging Adaptive Metrics to manage costs effectively. Vorwerk's experience highlights the benefits of open-source tools and the importance of customizing observability solutions to enhance operational efficiency and business insights.

Grafana Cloud has introduced several updates and new features across its managed observability platform, enhancing performance testing, frontend observability, Kubernetes monitoring, log management, synthetic monitoring, incident management, machine learning, and data querying. Notable additions include enriched test run summaries in GitHub Actions for Grafana Cloud k6, geolocation insights for frontend applications, role-based access control for frontend observability, and more efficient debugging tools for Kubernetes monitoring. The platform also now supports k6 extensions, allowing users to expand core functionalities and manage resources through Terraform. Additionally, Grafana Cloud has released new alerting options for synthetic monitoring, improved its IRM mobile app for incident management, and expanded machine learning capabilities with Anthropic model support. The Infinity data source plugin has been enhanced to support additional HTTP methods and gzip compression, improving data transfer efficiency. Furthermore, Grafana Cloud now offers cross-region connectivity using AWS PrivateLink, broadening its integration capabilities. These updates aim to provide users with more comprehensive tools for managing and optimizing their observability workflows.

In the realm of serverless computing, ensuring robust observability is vital, particularly for AWS Lambda functions. This text discusses the complexities and solutions associated with monitoring AWS Lambda using OpenTelemetry and Grafana Cloud. The challenges of serverless observability stem from Lambda's unique architecture, which lacks traditional servers, involves highly distributed components, and features ephemeral function executions. The text compares two approaches for instrumenting Lambda functions: using the AWS Distro for OpenTelemetry (ADOT) for straightforward setups with minimal configuration, and custom OpenTelemetry instrumentation for more complex requirements. It also outlines various data collection deployment options, including direct integration, agent deployment, and gateway deployment, each offering different levels of processing capabilities and flexibility. Additionally, the text highlights a practical project that tracks the Formula One "Driver of the Day" using AWS Lambda and Grafana Cloud, illustrating the application of these observability practices in a real-world scenario. Finally, it emphasizes the importance of understanding Lambda's execution lifecycle to ensure consistent telemetry data collection and invites feedback from users to enhance the observability solutions provided by OpenTelemetry and Grafana Cloud.

Embracing more incidents in incident response strategies can lead to improved systems by offering opportunities for learning and adaptation, despite initial skepticism about increasing incident frequency. By reframing what constitutes an incident, organizations can focus on uncovering insights about their systems, particularly the socio-technical aspects, without necessarily causing more disruptions. Elevating smaller or near-miss events to incident status allows for practice and coordination, essential for developing expertise and confidence in handling larger failures. The practice of declaring more incidents, supported by storing and analyzing related data, can enhance team resilience and readiness for significant challenges. This approach encourages a culture of openness and learning, ultimately strengthening incident management and system reliability.

SpotOn, a provider of restaurant point of sales systems, recently improved its observability strategy by migrating from multiple tools to Grafana Cloud and implementing a standardized tagging taxonomy. This change streamlined alerting and incident response, reduced costs, and enhanced data-driven insights. The tagging system, applied at both the infrastructure and application layers, categorizes assets by products/services, domain/teams, and system/components, utilizing tools like OpenTelemetry and Grafana Alloy. This new approach allows for efficient troubleshooting and cost management, with Grafana Cloud dashboards providing a unified view across business domains. The migration also fostered a cultural shift within the company, emphasizing collaboration and focusing on business metrics, ultimately transforming observability into a comprehensive team effort.

Many organizations are transitioning to Grafana Cloud IRM, a unified incident response and on-call management application, to enhance reliability, reduce costs, and streamline their tools. Grafana offers migration tools for legacy systems like PagerDuty, Splunk On-Call, and now Opsgenie, as Atlassian will discontinue Opsgenie sales in June 2025 and support by April 2027. The new Opsgenie migration tool facilitates the transfer of resources such as user notification rules, on-call schedules, escalation policies, and integrations to Grafana Cloud IRM, though some limitations in integration types and escalation policies exist. Grafana Cloud IRM has recently integrated its OnCall and Incident features into a single app, providing a unified homepage for alert groups and incidents, which enhances user experience and streamlines incident response. This consolidated approach allows for faster response times and better customization and automation within the Grafana Cloud observability stack. Grafana Cloud also offers a free tier and various plans to accommodate different user needs.

Grafana Labs has introduced a new serverless feature for integrating Microsoft Azure metrics and logs into Grafana Cloud, eliminating the complex configuration previously required with Grafana Alloy. This feature utilizes a newly developed Azure Prometheus exporter, which automatically discovers resources and metrics, and uses the Azure Monitor Metrics Batch API to avoid rate limiting errors. It provides enhanced insights not available through Azure Monitor, such as metrics on Azure Virtual Networks and Event Hubs. Users can now onboard with less than 10 lines of Terraform code, benefiting from a simplified setup process. The serverless capability also supports exporting Azure logs into Grafana Cloud, enhancing security monitoring and allowing for customizable data processing. Grafana continues to support users preferring to manage Alloy and plans to open-source the new Azure Metrics exporter, alongside other upcoming features to enhance its Cloud Provider Observability solution, which supports AWS, Azure, and Google Cloud.

Grafana Cloud has introduced Geolocation Insights within its Frontend Observability tool, providing real user monitoring (RUM) solutions that offer insights into the web application user experience through location-based data. This feature allows users to filter, segment, and analyze telemetry data by geographical regions, enabling optimization of web performance and error tracking while adhering to privacy regulations such as GDPR and CCPA. Geolocation Insights employs a privacy-first approach using an offline geolocation database to ensure data protection, offering anonymization, data minimization, and strict retention policies. To use geolocation tracking, both the Grafana Cloud UI and the Web SDK configuration must be activated, with the option to specify data granularity and exclude specific countries. The feature is designed to enhance understanding of frontend performance and user experience, supporting data-driven decisions while maintaining user privacy. Users can visualize geolocation data in Grafana Cloud dashboards, employing breakdown charts and interactive geographic visualizations to analyze regional trends and pinpoint issues.

Grafana Labs invites attendees to visit their booth at the Google Cloud Next 2025 conference, taking place from April 9 to April 11 at the Mandalay Bay Convention Center in Las Vegas. The event offers a platform for technology experts and practitioners to explore the latest in open source observability, including logs, metrics, traces, and application performance. Grafana Labs will showcase how users can centralize their data in Grafana Cloud for more efficient insights and cost savings, with opportunities to discuss Google Cloud Observability and the Cloud Provider Observability suite, which supports AWS, Microsoft Azure, and Google Cloud. Attendees are encouraged to schedule meetings with Grafana's experts for personalized recommendations and to share feedback on various open-source projects. Grafana Cloud offers a generous free tier and flexible plans to suit diverse use cases.

Grafana will fully remove support for AngularJS with the release of Grafana 12, following its deprecation in Grafana 9 and partial disablement in Grafana 11. Users relying on AngularJS-based plugins must migrate to React-based plugins to prevent disruptions, as AngularJS support will not be re-enabled. Grafana has provided tools and guidance to help users identify impacted dashboards and plugins, offering automatic migration options for some pre-installed visualization panels, while others require manual updates. Users are encouraged to plan their migration before the release to avoid issues, as any reliance on AngularJS will result in failed plugin loads and error messages. The Grafana community offers support through GitHub and forums for any migration challenges.

GrafanaCON 2025 will be held from May 6-8 in Seattle, offering an opportunity for attendees to connect with the open-source community, hear updates on OSS projects, and enjoy engaging activities. For those unable to attend in Seattle, GrafanaCON Local will bring the event's excitement to various cities worldwide, including Singapore, Berlin, Toronto, and Bangalore, throughout May. These half-day local events will feature the latest updates on Grafana and other tools like Prometheus and OpenTelemetry, enable networking within the local open-source community, and offer access to Grafana Labs experts for technical inquiries. Attendees are encouraged to register promptly due to limited seating, and further details are available on the GrafanaCON Local web page.

Grafana Cloud's latest update to its Kubernetes Monitoring solution introduces a new Storage tab designed to provide comprehensive visibility into Kubernetes storage volumes, enhancing the ability to track volume usage, conduct data forensic analysis, and troubleshoot provisioning issues. This feature offers a unified view of storage metrics, encompassing PersistentVolumes, PersistentVolumeClaims, and StorageClasses, through prebuilt panels and PromQL queries that visualize relationships between pods and volumes. The tab is accessible at various levels, from pod to cluster, offering insights into the capacity and usage of storage resources. Additionally, the update includes alert overlays from the kubernetes-mixin project, with alerts for volume filling and errors, helping users identify and address potential issues more efficiently. The integration of metrics from Prometheus exporters like kube-state-metrics, Kubelet, and cAdvisor further enriches the monitoring experience, with plans to incorporate file system metrics in future updates to optimize cost-effective storage class selection based on real-world data.