April 2024 Summaries

Grafana k6, an open-source performance testing tool, was created to address the limitations of existing solutions and to enhance software reliability through a developer-friendly platform. Since its inception, k6 has evolved to support both open source and managed cloud versions, providing engineering teams with a structured approach to performance testing. The blog post offers guidance on organizing performance testing suites with k6, emphasizing best practices such as modularizing test configurations, implementing reusable test scenarios, and creating object-oriented models for data access. It stresses the importance of treating performance testing like any other testing suite, starting simple, iterating, and focusing on performance regressions. Additionally, it highlights the need for error handling and observability to effectively troubleshoot issues, suggesting the integration of testing results with operational data for comprehensive insights. The post encourages sharing insights and best practices within the community to further refine and improve the use of k6 in performance testing.

Grafana Incident & Response Management (IRM) has introduced a feature that integrates Google Calendar out-of-office events with Grafana OnCall, aimed at ensuring seamless on-call shift coverage while providing engineers with peace of mind during personal time. This integration allows Grafana OnCall to access and monitor Google Calendar events to automatically generate shift swap requests when there's a scheduling conflict. Users can configure which on-call schedules are eligible for this feature and can manually request shifts if necessary. The integration is designed to simplify the on-call scheduling process, reduce stress, and enhance user experience, with additional features like web-based scheduling and a mobile app. This functionality is automatically available in Grafana Cloud, while users of Grafana OnCall OSS must follow specific steps to enable it.

GrafanaCON 2024, held in Amsterdam from April 9-10, showcased the latest developments in Grafana and its open-source ecosystem through technical talks and user success stories, now available on-demand for those who could not attend the sold-out event. The conference featured significant announcements including the introduction of Grafana Alloy, a distribution of the OpenTelemetry Collector, and the release of Grafana 11 with improved data connectivity and visualization capabilities. Key sessions included insights into Grafana Loki 3.0, which offers native OTel support and enhanced query acceleration, and the launch of the open-source Grafana App Platform designed to aid developers in creating complex applications. Noteworthy community stories highlighted Grafana's application in diverse fields such as CERN's data monitoring for the world's largest computing grid, Japan's lunar missions, and Planted's plant-based food production. The event also emphasized Grafana’s role in modernizing data systems at The National Library of the Netherlands, reflecting its versatility and impact across various industries.

The blog post details how to manage Grafana Cloud instances within a Kubernetes environment using the Grafana Operator, which allows for seamless integration of cloud services by managing them as Kubernetes resources. By using the Grafana Operator, users can configure folders, dashboards, and alerts directly within Kubernetes, maintaining all configurations in one place and adhering to the infrastructure-as-code principle. The post provides a step-by-step guide to installing the Grafana Operator via Helm and setting up external Grafana Cloud instances, including creating a service account and secret for authentication. It also explains how to create and manage resources like folders, dashboards, and alert rules, highlighting the benefits of this approach in simplifying workflows and increasing efficiency. The post encourages users to explore further resources like technical documentation, GitHub repositories, and community support channels to maximize the potential of Grafana Cloud and the Grafana Operator in Kubernetes environments.

At GopherCon 2023, Bryan Boreham, a distinguished engineer at Grafana Labs, presented a talk on optimizing merges using the loser tree data structure, which he implemented with Go generics to improve the performance of Prometheus and other open-source projects like Grafana Loki and Grafana Pyroscope. Boreham explained how loser trees, a type of tournament tree algorithm that stores the loser of each comparison, can accelerate k-way merges by quickly identifying the smallest element in a sequence, thereby enhancing execution times in systems that require efficient data merging. Despite the increased memory requirement, the implementation resulted in a performance boost, with merges being five to twelve times faster, as confirmed by benchmarking with Go's benchstat tool. Boreham shared his experience and enthusiasm for optimization through his work with loser trees, emphasizing the successful application of Go generics in improving data processing efficiency in large-scale systems. For further details, Boreham's full GopherCon session is available on YouTube, along with the code on GitHub.

Cloud computing, while offering numerous technical and operational benefits, is facing challenges in sustainability, with cloud data centers reportedly having a larger carbon footprint than the aviation industry. Ida Fürjesová and Niki Manoledaki from Grafana Labs addressed this issue at PromCon 2023 by showcasing a bottom-up approach to monitoring energy consumption and carbon emissions in cloud applications using tools like Kepler, Prometheus, and Grafana. Their session emphasized the concept of GreenOps, which prioritizes sustainability in cloud operations and can also lead to cost savings, enhanced app performance, and competitive advantages. Kepler, an eBPF energy monitoring tool, aggregates energy metrics and attributes power usage to processes and pods, exporting these metrics to Prometheus for visualization in Grafana. The session highlighted the potential for organizations to implement such practices to monitor and reduce their carbon footprint, and it encouraged participation in CNCF's Environmental Sustainability Technical Advisory Group for further collaboration and innovation in sustainable computing.

Grafana OnCall has released improvements for its Android mobile app to enhance the experience of on-call engineers by optimizing notification delivery, especially in overcoming the complexities of Android's Do Not Disturb (DND) modes. Unlike Apple's straightforward notification handling, Android's varied settings and OEM differences complicate ensuring critical alerts are both seen and heard. Previously, the app would adjust ringer modes and volume settings, inadvertently allowing all notifications to bypass silent mode, which was disruptive. The new approach utilizes the alarm sound channel, which bypasses silent mode without altering user preferences for other notifications, ensuring critical alerts are audible without unnecessary disturbances. This update, part of the 1.15.1 release, reflects Grafana's commitment to improving user experience and invites feedback from users to continue refining the app.

OpenTelemetry has become a crucial open-source tool for managing telemetry data, and this guide focuses on scaling the OpenTelemetry Collector across multiple Linux hosts using Ansible, enabling them to act as gateways and agents in observability architectures. By leveraging Ansible's ease of use and versatility, the guide simplifies the deployment, installation, and management of OpenTelemetry Collectors, which collect and forward metrics, traces, and logs to platforms like Grafana Cloud. The process involves setting up Linux hosts with SSH access, installing the Grafana Ansible collection, creating an inventory file, and configuring Ansible playbooks to deploy the OpenTelemetry Collector. Once deployed, users can verify data ingestion into Grafana Cloud and create custom dashboards to visualize the collected data, ultimately enhancing monitoring and observability capabilities. The guide provides insights on tailoring configurations to specific needs and highlights additional resources for further exploration of OpenTelemetry and Grafana's integration.

GrafanaCON 2024, held in Amsterdam, highlighted the latest developments in Grafana's open-source ecosystem, showcasing new releases, community use cases, and more through its keynote presentation. For those unable to attend the sold-out event, Grafana Labs offers a series of Keynote Recap webinars available in multiple languages and time zones, featuring prominent team members. These webinars will cover significant announcements including the release of Grafana 11, which introduces new visualizations and functionalities, enhancements in Loki 3.0 for improved log query performance and OpenTelemetry ingestion, and the launch of Grafana Alloy, a new open-source distribution of the OpenTelemetry Collector. Participants can attend live sessions or receive on-demand recordings, ensuring global accessibility to the latest Grafana updates.

Grafana Cloud's latest updates introduce a host of new features, including a queryless exploration of Prometheus metrics, native support for OpenTelemetry logs, and enhancements to the popular Canvas panel. At GrafanaCON in Amsterdam, attendees learned about improvements such as color-coded table rows, advanced flowcharting tools, and Scenes-powered dashboards for more dynamic interaction. The platform also unveiled new organizational tools like dashboard subfolders and user-based authentication for Azure Monitor, alongside significant upgrades to its PDF export functionality and label-based access control in Cloud Logs. Grafana Cloud now supports generative AI features and provides integrations for monitoring solutions like PgBouncer, OpenStack, InfluxDB, and SQL Server. As AngularJS support is phased out, Grafana encourages users to transition to React-based visualizations, offering tools to assist with this migration. These updates aim to enhance user experience, streamline data management, and support scalable observability solutions.

Grafana Alloy, a distribution of the OpenTelemetry Collector with Prometheus pipelines, supports metrics, logs, traces, and profiles, and can be efficiently deployed across multiple Linux hosts using Ansible. This guide explains how to leverage the Grafana Ansible collection, specifically the alloy role, to automate the installation and management of Grafana Alloy, enabling seamless data exploration in Grafana. Users are instructed to create an Ansible inventory file and a playbook that utilizes the alloy role, facilitating the sending of metrics from Linux hosts to Prometheus data sources. The guide also provides steps for verifying metric ingestion into Prometheus and troubleshooting potential issues, while suggesting the setup of clustering for enhanced performance and reliability. Additionally, Grafana Cloud offers a user-friendly platform for managing metrics, logs, traces, and dashboards, with a free tier available to users.

The article by Tobias Skarhed outlines a method for monitoring YouTube channel updates using Grafana, allowing users to receive alerts on new video releases without needing a YouTube account. By leveraging YouTube's RSS feeds, users can track updates from their favorite channels and playlists directly on their Grafana dashboard. Skarhed explains the process of obtaining channel or playlist IDs and configuring Grafana with the RSS/Atom feed data source plugin. Users can aggregate multiple feeds into a single panel using Grafana's "-- Mixed --" data source and transform the data for a cohesive display. While users cannot watch YouTube videos within Grafana, this setup provides a streamlined way to stay informed about new content from preferred channels.

Grafana Cloud has become a preferred choice for companies like SailPoint, Kushki, and Flexcity due to its ability to optimize telemetry storage, enhance observability strategies, and manage costs effectively. The companies experienced significant benefits from migrating to Grafana Cloud, backed by the expertise and support of the Grafana Labs team. Kushki achieved a unified monitoring solution by integrating multiple data sources, which improved their transaction success rate insights and response times. Flexcity utilized Grafana Cloud during a European energy crisis to monitor power consumption and production, benefiting from managed services and personalized support. SailPoint leveraged Grafana's Adaptive Metrics to reduce costs and manage their Prometheus deployment more efficiently, leading to a 33% reduction in metrics volume. Overall, these companies appreciated the technical advantages, transparency, and support provided by Grafana Labs, which helped them scale and explore new features with confidence.

Grafana Cloud provides a comprehensive observability platform that addresses concerns about cloud security by debunking common myths related to data protection in multi-tenant environments, integration with private data sources, and compliance with security standards. Despite initial skepticism about cloud-hosted platforms' ability to secure sensitive data, Grafana Cloud ensures data isolation and privacy through a robust architecture using tenant IDs for data separation and secure connections for private data source integration. It also maintains high security standards, evidenced by ISO 27001:2022 and SOC II Type 2 certifications, which are achieved through regular external audits. Additionally, Grafana Labs engages in proactive security measures, including a bug bounty program, to continually enhance its security posture. These efforts reassure organizations, leading them to migrate to Grafana Cloud to benefit from its extensive features and services while reducing operational risks.

Grafana 11, now available in preview, introduces several enhancements aimed at improving user experience, data visualization, and scalability. Key features include Scenes-powered dashboards, which offer a more stable and dynamic architecture, and Explore Metrics, a queryless feature for navigating Prometheus metrics. The Canvas panel has been upgraded with new flowcharting capabilities, while table visualizations can now have color-coded rows for better data interpretation. Grafana 11 also introduces AI-powered tools for generating titles and descriptions and improves alert management with features like 'Keep last state.' Additionally, the platform enhances organizational capabilities with subfolders for dashboards and user-based authentication for Azure Monitor data. The release also focuses on performance improvements, such as faster PDF exports and a better dashboard-editing experience. Grafana 11 is expected to be generally available by May 14, 2024, and users are encouraged to explore these features through Grafana Cloud or by downloading the preview version.

Explore Logs is a new open-source application designed to simplify log browsing without the need for Grafana Loki's query language, LogQL, providing an easier way for users to analyze their logs. Allowing users to explore logs through detected services, labels, detected fields, and patterns, the tool aims to reduce the complexity and cognitive load typically associated with traditional log querying. Users can take advantage of features like plain text search, pattern matching, and automatic service detection to streamline the process of identifying key insights and resolving issues faster. As part of Grafana's observability suite, Explore Logs integrates with Grafana 11 and Loki 3.0, offering a user-friendly interface for managing log data and enhancing the overall efficiency of site reliability engineers and developers. Available for preview, it invites feedback from users to further refine and improve the experience in collaboration with the community.

Grafana has transitioned from its Grafana Agent to a new open-source tool called Grafana Alloy, a distribution of the OpenTelemetry Collector that supports metrics, logs, traces, and profiles with native pipelines for OpenTelemetry and Prometheus formats. Grafana Agent, first introduced in 2020, addressed various complexities associated with Prometheus data management across different ecosystems but will now be deprecated, with Long-Term Support ending by October 2025. Alloy offers enhanced capabilities and simplifications, encouraging current Grafana Agent users to migrate to this new tool. The transition to Alloy is seen as a simplification of the observability stack, eliminating the need for multiple agents and modes, such as the Grafana Agent's Flow mode, and focusing future development efforts on Alloy. Users can expect comprehensive support and tools for migrating from existing Grafana tools or the vanilla OpenTelemetry Collector to Alloy, ensuring a smooth transition and access to the latest features.

Grafana Loki 3.0 marks a significant milestone in the evolution of the open-source log aggregation platform, introducing performance improvements and new features such as query acceleration with Bloom filters and native OpenTelemetry support. The addition of Bloom filters allows for faster and more efficient log searches by bypassing non-relevant data, while native OpenTelemetry ingestion simplifies the pipeline and enhances the querying experience without the need for data serialization. Alongside these features, the Loki documentation has been improved to enhance user accessibility, and quality-of-life improvements have been made to streamline usage, particularly through updates to the Helm chart. The successful development of Loki 3.0 highlights the community's contributions, with the release aligning with the upcoming Grafana Enterprise Logs 3.0, and users are encouraged to review release notes and documentation before upgrading.

Grafana Alloy, announced at GrafanaCON 2024, is the newest open-source distribution of the OpenTelemetry Collector, designed to integrate seamlessly with Prometheus telemetry formats and provide a comprehensive observability platform. Alloy builds upon Grafana Agent's existing infrastructure, offering compatibility with OpenTelemetry and Prometheus, while introducing enterprise-grade features like native clustering for scalability and Vault integration for enhanced security. As Grafana Agent transitions to long-term support and eventual deprecation, Alloy emerges as a robust alternative, facilitating the integration of various observability signals for infrastructure and application monitoring. Alloy supports a flexible configuration system, allowing it to be used in hybrid environments and paired with Grafana Cloud for enhanced application observability, Kubernetes monitoring, and out-of-the-box integrations. The tool is designed for scalability and ease of use with features such as GitOps-friendly configuration, debugging utilities, and fleet management for Grafana Cloud users, ensuring efficient management and monitoring of telemetry data across diverse IT environments.

GrafanaCON 2024 marked the return of Grafana Labs' annual conference in Amsterdam, showcasing major advancements in their open-source observability tools. The event featured a keynote unveiling Grafana 11, which offers innovations like a queryless experience for Prometheus metrics, enhanced dashboard functionalities, and improved alert rules. Concurrently, Grafana Loki 3.0 was introduced, enhancing log query performance with Bloom filters and supporting OpenTelemetry logs. A new project, Grafana Alloy, was launched as a 100% OTLP-compatible telemetry collector with enterprise-grade features. The conference also celebrated the 2024 Golden Grot Awards, recognizing exceptional Grafana dashboards, including Ruben Fernandez's commute-monitoring dashboard and Christopher Field's defect-monitoring project for steel alloys in nuclear reactors. The event concluded with the promise of further insights through recap webinars and additional detailed documentation available to the community.

ComplyAdvantage, a fintech company focused on compliance and risk management, enhanced its observability strategy by transitioning from on-premise Grafana OSS to Grafana Cloud, with OpenTelemetry playing a crucial role in the process. Adam Wilson, the principal SRE at ComplyAdvantage, shared insights at ObservabilityCON 2023 into how OpenTelemetry's vendor-neutral and open-source nature facilitated the migration, allowing the company to optimize its metrics, logs, and traces. The company deals with a complex infrastructure, including six billion spans daily and approximately 41 Kubernetes clusters, necessitating a robust observability stack. Initially struggling with the challenges of fragmented observability and overloaded Prometheus servers, ComplyAdvantage leveraged OpenTelemetry's capabilities to streamline its processes and improve data storytelling across the organization. The migration to Grafana Cloud enabled quicker integration due to native support for OpenTelemetry, fostering better alignment between technical teams. Post-migration, the company adopted Grafana SLO to manage service level objectives, finding a cultural fit with Grafana Labs that fostered a productive partnership. This transition not only improved operational efficiency but also allowed for impactful organizational insights, enhancing product and customer engagement.

Grafana is phasing out its legacy alerting system with the release of Grafana 11, urging users to upgrade to the more advanced Grafana Alerting system to prevent disruptions. Initially introduced with Grafana 9, Grafana Alerting provides a more powerful, scalable, and unified alert management system that is decoupled from dashboards, offering features such as Prometheus integration, advanced alerting rules, flexible notification management, dynamic templating, native as-code support, and high availability modes. Users can utilize the upgrade preview tool in Grafana 10.4 to transition smoothly, ensuring their alerts and configurations align with the new system before legacy alerting is completely removed. The changes affect Grafana OSS and Enterprise users, while Grafana Cloud users have already been upgraded. The new system enhances control and efficiency through single-pane management, customizable notifications, and robust access control, addressing community feedback and simplifying alert management across various environments.

The Prometheus community is actively integrating OpenTelemetry, an observability framework, to enhance its compatibility and reduce vendor lock-in, with significant steps taken to ensure bidirectional support between the two systems. This collaboration has led to the creation of specifications and implementations that allow seamless integration of Prometheus metrics into the OpenTelemetry Collector. As part of this initiative, Prometheus is preparing for the release of version 3.0 in 2024, which will feature OpenTelemetry Protocol (OTLP) support. The community is addressing challenges such as support for UTF-8 metric names, native resource attributes, and Delta temporality handling, while striving to ensure these updates do not disrupt existing Prometheus operations. The Prometheus team is calling for contributors to help with these developments and has structured efforts to make the maintainer process more accessible. This marks a significant evolution for Prometheus, promising enhanced interoperability and functionality without major disruptions for current users.

Grafana Alerting has introduced a new simplified routing feature designed to retain the power of its original, complex routing system while making it more accessible and user-friendly. Previously, Grafana Alerting employed a tree-structured routing approach using notification policies and labels, which, despite being powerful, proved too complex and confusing for many users. The new system allows users to select a contact point directly, automatically generating a policy in the background, thereby eliminating the need to manually configure labels and resulting in a more intuitive user experience. This update, while simplifying the process, maintains essential features like modifying policies, muting timings, and changing groupings, and it enhances control through alert rule Role-Based Access Control (RBAC), reducing errors from unauthorized changes. Simplified routing is currently available for Grafana managed alerts and Alertmanager, with options for customization and new contact point creation accessible to those with appropriate permissions. Auto-generated policies are visible only to admin users, who can manage them through the alert rule form, ensuring that the system remains secure and organized.

Brian LeBlanc, a Senior Solutions Engineer at Grafana Labs, devised a creative solution to monitor his sons' chores using Grafana Cloud, developing what he calls "Roboparenting." He set up a system to track his children's room-cleaning activities and send notifications via Slack if they failed to meet expectations. By creating a custom exporter for Prometheus and integrating it with Grafana Cloud, LeBlanc was able to visualize the data and generate alerts for his sons, which are then routed to separate Slack channels. He also implemented functionality for conditional logic, snooze notifications, and custom metrics to ensure the system adapts to different times and scenarios. The experiment has been a fun engagement for LeBlanc's tech-savvy children, who, alongside their parents, view it as an effective way to manage household chores while offering the potential for future enhancements like performance-based incentives.