May 2024 Summaries

Ruben Fernandez, an SRE from Atlanta, developed an innovative Grafana dashboard that won the Golden Grot Award for its practical application in improving his daily commute. His dashboard aggregates real-time data from various sources such as weather.gov, the Metropolitan Atlanta Transit Authority, Google Maps, and Bing Maps to help him decide on the best time and mode of transportation for his journey to work. The initiative stemmed from his need to return to the office post-pandemic, prompting him to consider traffic, weather, and commute duration in a single, accessible platform. Despite the challenges of creating the dashboard as a side project, Ruben's determination and creativity were recognized and celebrated by his family, friends, and the judging community. His work showcases how personal dashboards can transform everyday tasks, offering a futuristic approach to decision-making for daily commutes.

Grafana dashboards are powerful tools for visualizing technical data, but exporting this data for further analysis or sharing is often necessary. This guide provides a detailed walkthrough of exporting data from Grafana visualizations into CSV files, which can then be imported into spreadsheet applications like Microsoft Excel or Google Sheets. The process works universally for any Grafana visualization and data source because all visualizations are rendered as data frames. Users can export data from a Grafana dashboard or Grafana Explore queries by selecting the desired data frame and format, such as CSV, and downloading it, complete with headers for easier integration into spreadsheet applications. The guide also emphasizes the convenience of joining multiple data series by time for comprehensive data exports and highlights the ability to apply these steps to both dashboard visualizations and exploratory queries.

Grafana Loki has introduced a new feature in its 3.0 version called query acceleration, designed to speed up log queries without requiring additional computational resources. By leveraging n-grams and Bloom filters, this experimental feature aims to reduce the amount of irrelevant data processed during queries, enhancing efficiency and maintaining cost-effectiveness. Bloom filters, which allow for probabilistic matching with some false positives but no false negatives, help identify and skip non-matching data, thereby increasing query speed significantly. The new system also includes a query sharding strategy that uses Bloom filters to create fewer, fairer query shards, addressing uneven data distribution in traditional sharding. Although Bloom filters are especially effective for precise searches, not all queries will benefit from them, and certain conditions must be met for their application. This development stays true to Loki's principles of being schemaless and easy to use, with no additional configuration required for users when it becomes available on Grafana Cloud.

Grafana OnCall has introduced a new bi-directional integration with ServiceNow, enhancing the incident management process for organizations using both platforms. This integration allows alerts to seamlessly flow between Grafana OnCall and ServiceNow, ensuring incident statuses remain synchronized across systems. The feature is configurable, enabling users to adjust ServiceNow credentials and fine-tune alert behavior according to their specific needs. Incidents created in ServiceNow can automatically generate alert groups in Grafana OnCall, which are then routed through established mechanisms to alert the appropriate teams. The integration also supports mapping custom incident statuses from ServiceNow to Grafana OnCall for consistent status updates. Additionally, users can create ServiceNow incidents based on alerts from various tools within their observability framework, centralizing incident management. Customization options are available through Grafana OnCall's outgoing webhooks, allowing teams to tailor the integration to their incident response workflows.

Christopher Field, co-founder and president of Theia Scientific, has been recognized with the Golden Grot Award for his innovative use of Grafana dashboards in monitoring steel alloys for next-generation nuclear reactors. His company developed a system that streams images from electron microscopes into a time series database and machine learning models, allowing researchers to quickly identify defects in alloys exposed to radiation. This approach has significantly reduced the time required for analysis, transforming a process that used to take years into one that can be completed in real-time. This innovative application of Grafana highlights its versatility beyond traditional IT and DevOps uses, demonstrating its potential in scientific research by facilitating faster, more efficient decision-making. Field's advice to new users of Grafana is to engage with the community, start small, and not fear initial failures, emphasizing that experimentation can lead to powerful, effective solutions.

OpenTelemetry has become a vital open-source tool for collecting telemetry data, such as traces, metrics, and logs, offering users more control by reducing dependency on proprietary software. This article explores how to use OpenTelemetry to monitor Google Cloud Run, a serverless solution in Google Cloud Platform, by utilizing auto-instrumentation with a Node.js application. Google Cloud Run's recent architectural update allows multi-container deployments, enabling improved monitoring and logging through sidecar containers like the OTel Collector, which gathers telemetry data. The data is then sent to Grafana Cloud Application Observability, which provides visualization and analysis through pre-built dashboards and the Rate Error Duration (RED) method, facilitating anomaly detection and performance optimization. The process involves setting up Node.js applications with auto-instrumentation, configuring the OTel Collector, creating Dockerfiles, and deploying services on Cloud Run, ensuring the OTel Collector starts before application containers to avoid data loss. This integration of OpenTelemetry and Grafana Cloud enhances observability in serverless environments, allowing teams to proactively identify issues and optimize application performance.

Kyle Eckhart's blog post explores various methods for visualizing Amazon CloudWatch metrics using Grafana, emphasizing flexibility and user choice. The post outlines four main approaches: using the Grafana Labs Amazon CloudWatch data source, employing the AWS Observability app, running Grafana Alloy with the CloudWatch exporter, and using the yet-another-cloudwatch-exporter (YACE). Each method utilizes Amazon CloudWatch APIs to access data, supporting a wide range of CloudWatch namespaces, including custom metrics. While the Amazon CloudWatch data source queries data directly from AWS, the YACE-based options extract and transform metrics into Prometheus-compatible formats for centralized storage, enabling cross-account, cross-region, and cross-service dashboard capabilities. Despite their similarities, these methods differ in storage, fidelity, and operational complexity, with YACE-based options offering features like "Resource Tag Association" for enhanced data insights. The blog stresses that Grafana Cloud is a user-friendly starting point for AWS observability, offering a free tier and future developments like metric streaming support and OpenTelemetry integrations.

Grafana Beyla has been integrated into Grafana Alloy as the default eBPF-based auto-instrumentation solution, enabling users to automatically capture metrics and traces from running services and connect to existing telemetry pipelines. Grafana Alloy, an open-source distribution of the OpenTelemetry Collector, supports OTLP-compatible pipelines for various telemetry formats and has recently been updated to version 1.1, featuring Grafana Beyla 1.5. With this integration, users can obtain RED metrics without modifying their code and automatically instrument applications within Kubernetes environments using Helm charts. The blog post details the configuration steps for both scenarios, emphasizing the ease of deployment and potential for customization. Future updates are expected to include more configuration options for Beyla, support for additional protocols, and enhanced eBPF-based features like network monitoring.

Grafana OnCall has expanded its webhook support to allow integration with messaging services like Mattermost and Discord, enabling users to receive alert notifications and updates in these platforms. The integration involves creating bots that post details of alert groups to designated channels, keeping the status and information up to date through Grafana OnCall webhooks. For Mattermost, users must create a bot account to access the API and configure webhooks to post and update messages based on alert group activity, while for Discord, users can set up webhooks to create and edit messages using a similar process. The guide also mentions the potential for extending this approach to other services like Microsoft Teams and Google Chat, and offers script resources for programmatically setting up webhooks with the Grafana OnCall API. The text concludes by promoting Grafana Cloud, highlighting its ease of use and free tier offerings.

Grafana Cloud has introduced a series of updates aimed at enhancing its observability platform, powered by the open-source Grafana LGTM Stack, which includes tools like Loki for logs and Grafana for visualization. The latest updates feature significant revamps to Synthetic Monitoring, now incorporating Grafana k6 for more comprehensive API and web application performance monitoring with MultiHTTP and k6 Scripted checks, which facilitate complex transaction testing without the need for extensive coding. Kubernetes Monitoring has also been improved with user experience enhancements such as customizable refresh intervals and Sift investigations for infrastructure telemetry. The platform now supports monitoring Velero and OpenLDAP through new integrations, and additional Kubernetes integrations for TensorFlow, OracleDB, Microsoft SQL Server, and Istio. Furthermore, Grafana SLO updates allow users to filter service level objectives by team and service, aiding in performance tracking, while the Alerting Provisioning HTTP API now enforces Role-Based Access Control for better security management. These updates are part of Grafana Cloud's ongoing efforts to provide a robust, fully managed observability platform with a generous free tier for new users.

ObservabilityCON 2024, hosted by Grafana Labs, is slated for September 24-25 in New York City, marking its annual flagship open-source observability conference. The event will offer attendees a chance to hear the latest updates on the Grafana LGTM Stack, participate in hands-on workshops and technical sessions, and engage with the community through live demos and an "Ask the Experts" booth. Registration opens at the end of June, with early bird pricing available for a limited number of tickets, and opportunities for speaking and sponsorship are currently open. Additionally, a condensed version of the conference will travel to Berlin, London, Paris, and São Paulo later in the year, extending the reach of this influential event.

Grafana has introduced Flame graph AI, an AI-powered tool designed to enhance the interpretation of flame graphs within Grafana Cloud Profiles, a continuous profiling platform. This feature utilizes a large-language model (LLM) to help developers identify bottlenecks, root causes, and suggest fixes more efficiently, addressing the challenge of turning profiling data into actionable insights. During a Grafana Hackathon, the AI system demonstrated superior performance in interpreting flame graphs compared to human users of varying expertise levels. The tool, now generally available, integrates with GitHub to provide line-level insights and targeted recommendations by linking flame graph nodes directly to the source code. This integration aims to streamline the problem-solving process by offering precise analysis and advice on code optimizations, enhancing the overall usability of developer tools.

Grafana's powerful transformation capabilities allow users to manipulate and visualize data from various sources within a single platform, enhancing the efficiency and flexibility of data dashboards. These transformations enable users to refine data without altering the original source, perform joins, and execute mathematical operations, while also allowing for complex workflows through sequential application. Recent updates aim to improve the user experience with a new UI that facilitates browsing and understanding transformations, enhanced debugging tools, and performance gains by reducing redundant data fetching. Grafana 11 introduces features such as improved date and time formatting, new hierarchical data visualization options, substring matching for data filtering, and unary operations for field calculations. The platform also supports advanced customization for threshold colors, dashboard variables, and enum value plotting in time series visualizations. Additionally, Grafana offers comprehensive documentation and community engagement opportunities to help users maximize their experience with the latest features.

Grafana 11 introduces a range of new features centered around the Canvas panel, enhancing its capabilities for creating custom visualizations within Grafana's UI. The Canvas panel allows for dynamic flowcharting, enabling users to build intricate connections and customize their visual elements with features like color, size, and animation. Interactive buttons can now be added to dashboards to control external systems via API endpoints, transforming Grafana into a bidirectional monitoring solution. Improvements in ease of use include pan and zoom features for large canvases, snapping and alignment tools, and expanded data link support. These enhancements aim to provide users with increased control and flexibility in visualizing, managing, and interacting with their data. Grafana 11's features are available in both its open-source download and Grafana Cloud, with step-by-step upgrade guides and community support forums to facilitate the transition and encourage user engagement and feedback.

Grafana 11 introduces an innovative feature called Explore Metrics, designed to simplify the process of browsing Prometheus-compatible metrics without the need for PromQL queries. Demonstrated at GrafanaCON 2024, this feature aligns with Grafana's long-standing mission of enhancing observability through user-friendly design and flexibility. Explore Metrics, currently in public preview, allows users to navigate and analyze metrics with ease, offering functionalities such as automatic visualization selection, related metrics insights, and a user step history for process reference. This advancement reflects Grafana's commitment to making data exploration more dynamic and intuitive, especially during critical incidents when quick insights are essential. The release of Grafana 11, available for general use, also brings a suite of new features aimed at improving usability and extending the platform's capabilities. Users are encouraged to explore these updates, engage with the Grafana community, and provide feedback to continue the platform's evolution.

Grafana Alerting introduces a suite of updates aimed at enhancing incident resolution speed and reducing alert fatigue, significantly impacting infrastructure reliability and engineering productivity. With the latest iteration, Grafana Alerting allows users to alert on logs and metrics from any data source, offering improved management and triage capabilities. Key updates include simplified dynamic alert routing, an overhauled alert details view for expedited triage, reduced total cost of ownership through optimized rule evaluations, and the introduction of a "keep last state" feature to minimize noise from intermittent alerts. Enhanced security measures, such as role-based access control for provisioning APIs, and tighter integration with Grafana OnCall are also featured, alongside distributed evaluation intervals to ensure consistent performance and reduced resource spikes. Furthermore, users can now set recovery thresholds for alerts, enabling more nuanced observability strategies. These enhancements, part of the broader Grafana 11 release, are designed to improve user experience and operational efficiency, with options for seamless upgrades and community engagement through Grafana Labs forums.

Grafana Enterprise data source plugins are a key feature of Grafana, enhancing its role as an open and extensible observability platform by allowing users to connect and visualize data from various external systems through Grafana dashboards. These plugins enable users to view and correlate metrics, logs, and traces from multiple sources without data migration, thereby improving troubleshooting and reducing Mean Time to Resolution. Grafana Enterprise data sources, built and supported by Grafana Labs, are available in Grafana Enterprise and Grafana Cloud, with a variety of pricing models depending on the tier and number of active users. The plugins support connections to proprietary tools and platforms like Splunk, Dynatrace, and Databricks, providing a seamless user experience through easy query building and data filtering. Installation and configuration are straightforward, especially in Grafana Cloud, where a one-click installation process is available. Users can access additional resources and documentation to learn more about these plugins, with options for free-tier access in Grafana Cloud for a limited number of users.

Grafana Alloy, a distribution of the OpenTelemetry Collector with built-in Prometheus pipelines, offers simplified collection of Microsoft Azure metrics by enabling subscription-level data gathering and consolidating configurations. Traditionally, Azure metrics were collected resource-by-resource, which could lead to inefficiencies and API rate limiting due to the high volume of resource calls. Grafana Alloy addresses these issues by allowing users to collect metrics from all relevant resources in a region with a single API call, thus reducing the number of API calls significantly. Additionally, it streamlines configuration management by supporting all necessary dimensions in one configuration, eliminating the need for separate setups for different metrics and their dimensions. This advancement not only minimizes the complexity and operational overhead but also ensures high availability and scalability through Alloy clustering. Grafana Labs encourages users to leverage these features for improved observability of their Azure workloads and offers support to assist users throughout their observability journey.

AWS users can now stream their logs to Grafana Cloud more cost-effectively with the newly launched "Logs with Firehose," which leverages Amazon Data Firehose for real-time log streaming. This approach is simpler and cheaper compared to the traditional "Logs with Lambda" method, as it avoids the high costs associated with per-invocation charges by billing based on data volume. The Firehose solution is designed to facilitate the connection of Amazon CloudWatch logs to Grafana Cloud without the need for agents or local configurations, offering a seamless integration with AWS services such as VPC Flow Logs and AWS CloudTrail. Users can also back up logs to Amazon S3, enhancing data reliability. Although Firehose is not suitable for all scenarios, such as load-balancer access logs, it provides a more economical solution for streaming high volumes of log data. Grafana Cloud, combined with the AWS Observability app, offers a comprehensive view of AWS environments, enabling users to monitor their infrastructure and application telemetry efficiently.

A recent security report inaccurately claimed a SQL injection vulnerability in Grafana, but Grafana clarified that the behavior in question is intended and documented for authenticated users. Grafana's design allows authenticated users to have the same permissions as the user configured for the data source, which is central to its "big tent" strategy and performance. Grafana supports data visualization from numerous data stores without copying data, maintaining the original data as the source of truth. Users can access data through anonymous or authenticated modes, with authenticated users categorized as Viewer, Editor, or Admin, reflecting their permissions with the data source. Grafana emphasizes the importance of configuring data source credentials carefully, especially with SQL data sources, to prevent unauthorized data alterations. Administrators are encouraged to regularly review and limit data source credentials to prevent potential security breaches. Public dashboards and the Explore feature offer ways to limit query capabilities, and users can seek support through Grafana Labs or community forums. Grafana Labs also provides a process for reporting and managing security vulnerabilities, advising users not to disclose vulnerabilities until they are addressed.

Grafana Labs is enhancing its incident response and management tools to improve the way teams handle incidents, focusing on faster and simpler solutions through Grafana Incident, which leverages machine learning and integrations for efficient problem resolution. Innovations include Sift, a diagnostic assistant that uses machine learning for quick issue prioritization, and improved collaboration via Grafana OnCall, which ensures that appropriate team members are notified promptly during incidents. The integration of OpenAI helps generate concise incident summaries to aid documentation, while communication features like Slack Attachment Uploads enhance information sharing. Future enhancements involve customizable incident phases, private incidents to enhance security, and streamlined Slack integration for better user experience. The updates aim to maintain operational continuity, reduce system downtime, and align incident management with internal workflows, offering mobile capabilities for incident management on the go and automatic incident declaration as part of escalation processes.

Pepe Cano's blog post provides a comprehensive guide on organizing a performance testing suite using Grafana k6, emphasizing the importance of utilizing JavaScript tools like ESLint and Prettier for code quality and the strategic setup of repositories and shared libraries for collaboration. The post explores the debate between monorepos and multiple testing suites, recommending alignment with existing project structures to enhance team involvement and streamline performance testing integration. It highlights the utility of JavaScript bundlers such as Rollup and Webpack for managing shared libraries and supporting TypeScript, which adds type safety for larger projects. The post also underscores the role of a central team in maintaining shared libraries and best practices, while encouraging community engagement to further refine testing methodologies.

Grafana Labs has enhanced its Grafana Cloud Synthetic Monitoring service to simulate complex user journeys and transactions, aiming to improve end-user experience by utilizing the capabilities of Grafana k6. This evolution includes the introduction of two new check types, MultiHTTP and Scripted, which allow users to create multi-step tests and define workflow tests in JavaScript, respectively. These new features enable developers, platform engineers, and site reliability engineers to monitor complex transactions beyond protocol-level tests, which were previously limited to the Prometheus blackbox exporter. The integration of k6 offers a robust solution with a high-performance JavaScript engine and a scripting API, allowing for comprehensive synthetic monitoring by simulating real application traffic. Moreover, it supports HTTP, WebSockets, and gRPC protocols, and includes built-in metrics and custom logs for troubleshooting. Users can automate the setup and maintenance of checks, activate alerts, and visualize results through built-in dashboards. The service is designed to be adaptable for load testing and performance insights, promising further enhancements, such as the incorporation of the k6-browser module, to expand its capabilities. Grafana Cloud offers a free tier with significant test executions, making it accessible for users to start with synthetic monitoring.