March 2024 Summaries

Grafana OnCall has been enhanced to offer seamless integration with the rest of the Grafana Cloud ecosystem, aiming to improve incident response and management workflows. The tool now integrates with other components of the Grafana LGTM Stack, such as Loki, Grafana, Tempo, and Mimir, to provide a cohesive experience. New features include the ability to automatically run Sift, a machine-learning-powered diagnostic tool, when a new alert group is created, which helps in quickly diagnosing issues by analyzing logs, metrics, and traces. Additionally, improvements have been made to streamline the process of connecting Grafana OnCall with Grafana Alerting, and users can now easily declare incidents directly from Grafana OnCall or its dashboards, reducing the time and context switching needed for incident management. The platform also allows for easy paging of users or teams from an active incident, enhancing coordination across teams. These updates are part of a broader effort to enhance the user experience and operational efficiency within the Grafana Cloud environment.

Designing an effective incident response process is crucial for maintaining service availability in modern software operations, which often rely on numerous external dependencies and cloud services. This guide emphasizes the importance of defining what constitutes an incident, establishing robust alerting systems, and classifying incidents based on severity to prioritize responses appropriately. The roles of the incident commander and investigator are essential for managing incidents, with the commander supporting and communicating throughout the process while the investigator focuses on resolution. The post-incident phase involves creating a post-incident review to document and learn from the event, as well as establishing follow-up tasks to address root causes. The article suggests using tools like Grafana IRM to streamline incident management and encourages organizations to tailor their response processes based on their specific needs and contexts.

In a recent exploration of CI/CD observability, Grafana Labs team members, including Horst Gutmann and Dimitris Sotirakis, extended their previous work on continuous integration by delving into continuous delivery (CD) to extract DevOps Research and Assessment (DORA) metrics. This initiative aimed to create a proof-of-concept for gathering DORA metrics from internal services' CD workflows, with a focus on velocity and stability. Using tools like Argo Workflows and Grafana Incident, the team designed a data pipeline to collect and visualize metrics such as deployment frequency, lead time for changes, change failure rate, and mean time to recovery. They utilized OpenTelemetry for standardizing and sharing telemetry data across different platforms, highlighting its potential for broad adoption in CI/CD observability. The project underscored the value of these metrics in optimizing software delivery processes and identified potential areas for future exploration, including greater integration with logs and contributions to the CNCF CI/CD Observability working group.

OpenAI's ChatGPT Vision API, also known as GPT-4 with vision (GPT-4V), can be leveraged to automate image analysis, transforming a traditionally manual task into an efficient, automated process. The example provided in the text demonstrates how to use this capability to count vehicles entering Yellowstone National Park using images from a security camera feed. The process involves initializing the OpenAI API client, preparing vision prompts for the API, and then having ChatGPT analyze the images to identify and count vehicles. The results are formatted in JSON and pushed to Grafana Cloud Metrics using the Influx Line Protocol, allowing users to visualize the data in real-time. This integration not only enhances efficiency but also opens up possibilities for various unique use cases by combining computer vision and metric visualization technologies.

Grafana Labs has introduced new features in Grafana 10.0 that enhance data visualization and trend analysis, enabling users to better interpret large datasets. These include sparklines in table visualizations, regression analysis for time series data, and trend panels that allow for continuous data visualization beyond time-based metrics. The regression analysis transformation offers options such as linear and polynomial models to help smooth out data extremes and better understand long-term trends. Stat panels now feature a percent change indicator to provide insight into metric trends. Additionally, Grafana Cloud enhances these capabilities by offering metric forecasting for future trend assessment, positioning itself as a comprehensive tool for managing metrics, logs, traces, and dashboards with various subscription plans, including a free tier.

Monitoring identity provider logs is crucial for identifying potential security threats, and Sigma rules offer a standardized way to define pattern detection rules in YAML format, enhancing the capability to search logs efficiently. The Grafana Labs Security Operations team has developed a GitHub Action that automates the validation of Sigma rules using a JSON Schema they created, replacing the outdated Rx Schema previously in use. This initiative was part of an internal project that integrated pySigma and pySigma-backend-loki, which translates Sigma rules into LogQL queries for more effective security monitoring. The JSON Schema is now available on the Schema Store and is used to validate each new commit in the Sigma repository. The GitHub Action, now available on the GitHub Marketplace, allows users to validate Sigma rules in both public and private repositories, accommodating custom JSON Schemas if needed. This collaborative project with the Sigma maintainers has resulted in the creation of the sigma-rules-validator project, which encourages community feedback and contributions to further enhance the tool's functionality.

Shipping and third-party logistics companies, such as DHL, Blinkit, and Maersk, have dramatically improved their operational efficiency and customer satisfaction by implementing the Grafana LGTM Stack, which includes tools like Grafana for visualization, Loki for logs, Tempo for traces, and Mimir for metrics. DHL Express enhanced its monitoring systems by transitioning from a reactive approach to a proactive one, utilizing Grafana dashboards and alerting systems to quickly resolve issues and reduce stress for its IT team. Blinkit replaced its ELK Stack with Grafana Cloud to streamline real-time data access, crucial for maintaining an efficient checkout process that directly impacts customer satisfaction and sales. Maersk, responsible for a significant portion of global trade, adopted a centralized observability platform using the Grafana stack, enabling self-service onboarding and real user monitoring through Grafana Faro, thereby improving reliability and agility in its operations. These implementations illustrate how the Grafana LGTM Stack supports logistics companies in maintaining smooth operations and enhancing customer experiences by providing comprehensive visibility and reducing mean time to recovery (MTTR).

Grafana Beyla 1.3 enhances its OpenTelemetry distributed tracing capabilities using eBPF auto-instrumentation, building on previous versions that were limited to single span traces. The tool now implements two approaches: automatic header injection and black-box context propagation, aiming to provide comprehensive distributed tracing. Automatic header injection uses eBPF probes to read and inject trace information into outgoing calls, currently supporting Go applications, though it has limitations when the Linux kernel is in lockdown mode. Black-box context propagation leverages TCP connection tuples to track interactions between services, independent of programming language, but is currently limited to single-node operations. Future versions of Beyla aim to extend support for more languages and overcome existing constraints, such as those related to asynchronous requests in non-Go applications. The developers invite community feedback and participation through Slack channels and monthly community calls, offering resources like technical documentation and tutorials for further exploration of Beyla's capabilities.

Integrating monitoring and logging into application stacks is essential for maintaining performance, enhancing security, and troubleshooting, especially when using Grafana Cloud to collect metrics and logs in environments where traditional pull-based methods are not feasible. This is particularly relevant in ephemeral and serverless architectures, batch jobs, IoT systems, and CI/CD workflows, where the ability to push data directly through an HTTP API to Grafana Cloud ensures real-time insights into application performance. The guide details how developers can configure metrics and logs to be sent directly using an HTTP API, providing examples in various programming languages and emphasizing the strategic advantages of this approach over traditional agent-based methods. However, it also notes that in stable, long-running environments, traditional pull-based systems like Prometheus might offer more comprehensive monitoring solutions. By leveraging these HTTP APIs, organizations can enhance their observability strategy, achieving a detailed view of system health while facilitating informed decision-making and streamlined operations.

Grafana Cloud, a managed observability platform, has introduced several updates to enhance user experience and functionality. Notable enhancements include the Canvas panel's improved alignment features for custom visualization, new styling options for the geomap geojson layer, and streamlined data presentation in table panels. Additionally, significant advancements in log monitoring have been made with the Log Volume Explorer for cost management and Cloud Logs Export for extended log retention. The integration of AI/ML features, such as flame graph AI for profiling insights and AI-generated dashboard descriptions, offers users faster data interpretation. Simplified alert routing improves notification management, while experimental features like TraceQL metrics allow for dynamic metrics creation from traces. New solutions for AWS Observability, Apache Solr, and Windows Active Directory further extend Grafana Cloud's monitoring capabilities. As AngularJS support phases out, users are encouraged to transition to React-based plugins to ensure seamless dashboard functionality.

Grafana Agent v0.40 introduces enhanced customization options with the addition of custom components and a revamped module configuration system, aimed at simplifying the configuration process for users. Custom components allow users to create reusable components by utilizing existing elements and the standard library, which is particularly useful for tasks like reusing specific URLs across different pipeline stages. The release also includes Modules 2.0, which facilitates configuration reuse among multiple pipelines, especially in environments like Kubernetes, by enabling standardized labeling across applications and infrastructure. This update supports remote updates via import functions, allowing Grafana Agent to automatically load changes, and encourages community feedback through Grafana Labs Community Slack and the Agent repository. Grafana Cloud offers an accessible entry point for users with a free tier for managing metrics, logs, and more.

AWS Observability in Grafana Cloud introduces a new, user-centric application designed to simplify and enhance the monitoring of AWS environments. Built on the open-source Yet Another CloudWatch Exporter (YACE), the app offers a more integrated and intuitive experience by combining resource tags with CloudWatch metrics, allowing for flexible and powerful data querying without additional infrastructure. Key features include agentless operation, cross-account and cross-regional monitoring, and dedicated dashboards for Amazon EC2 and other AWS services, enabling users to optimize resource management and decision-making. The app incorporates new capabilities, such as accessing custom namespaces and tagging metrics to streamline data filtering, along with retaining popular dashboards for cost, storage, and service performance insights. Future updates aim to address specific user challenges with features like an RDS view for database efficiency and log streaming for real-time data analysis, positioning the app as a comprehensive tool for AWS monitoring.

Grafana Labs is set to participate as a silver sponsor at KubeCon + CloudNativeCon Europe 2024, a major event for open-source enthusiasts, taking place from March 19 to March 22 at Paris Expo Porte De Versailles. The team will be involved in several sessions covering topics such as using eBPF for instrumenting managed runtime applications, debunking myths about environmental sustainability in the cloud, addressing challenges in monitoring asynchronous workflows, and strategies for rebuilding cloud-native communities. Notable presentations include discussions on Prometheus updates, scaling metric collection with Kubernetes Operators, and the experiences of Grafana's development team in enhancing deployment processes for a large-scale application. These sessions aim to provide insights into various technical challenges and innovations in the cloud-native landscape, alongside opportunities for attendees to engage with Grafana Labs experts and explore new developments in observability and sustainability.

The on-call experience for software engineers at Grafana Labs, as shared by senior engineers Michael Mandrus and Owen Smallwood, is not as daunting as commonly perceived, and can even be rewarding and enjoyable. Both engineers, who joined Grafana Labs in 2022 and began their on-call rotations in 2023, describe the initial challenges of adapting to quick decision-making and learning new tools but emphasize the importance of collaboration and communication during these shifts. They highlight that the role is more about maintaining the system’s functionality than knowing everything, and the experience is enriched by the support of a collaborative on-call community. The engineers also discuss practical strategies such as taking breaks, customizing alert notifications, and seeking help when needed. Grafana Labs encourages shadowing programs and team ownership of projects, which has shifted their on-call culture toward more integrated team responsibility. The experience has allowed them to gain deeper insights into the infrastructure and environment of their applications, contributing to their growth as engineers.

Grafana Labs introduces the grafana/otel-lgtm Docker image, a preconfigured OpenTelemetry backend aimed at facilitating open-source monitoring and integration testing. This image integrates with the OpenTelemetry Collector, Prometheus, Loki, Tempo, and Grafana, allowing users to collect, store, and visualize telemetry data such as metrics, logs, and traces. It supports both gRPC and HTTP protocols for receiving OpenTelemetry signals and provides example Grafana dashboards for quick setup. Although designed for development and testing environments, grafana/otel-lgtm simplifies the process of running integration tests by enabling users to verify telemetry signals through Grafana’s HTTP API. The image is open-source, available on Docker Hub, and supports community contributions, with plans for further enhancements and integration with projects like Quarkus.

Grafana Labs' 2024 Observability Survey, with input from over 300 practitioners, highlights the evolving state of observability practices across industries. While mature practices help organizations respond to incidents swiftly and cost-effectively, many teams still operate reactively, often relying on customer feedback to identify issues. The survey underscores the complexity faced by organizations due to the widespread use of multiple observability tools and data sources, with centralized observability proving time and cost-efficient for many. Open source tools like Prometheus and OpenTelemetry dominate the landscape, with significant overlap in their adoption. Although the role of AI in observability is currently limited, practitioners are optimistic about its future potential to enhance incident response and streamline adoption, particularly in anomaly detection. Cost concerns, including cardinality and vendor lock-in, remain prominent, reflecting the ongoing challenges in the observability field.

Grafana 11, set for release in May, will disable AngularJS support by default across all instances, potentially affecting plugins reliant on this deprecated framework. This change is part of Grafana's transition from AngularJS to React, which began with Grafana v5 and has been progressing through subsequent versions. Affected users are advised to check for any AngularJS dependencies through in-product warnings and Grafana's documentation, and to migrate to React-based plugins or suitable alternatives to ensure uninterrupted use. Grafana has introduced tools and resources, including an open-source tool to detect AngularJS dependencies and a developer portal for migration guidance. The removal of AngularJS aims to streamline plugin development and maintenance, although it may cause temporary disruptions. Grafana's plugin catalog continues to grow with new offerings, and users are encouraged to prepare for the full removal of AngularJS support in Grafana 12.

Profile-guided optimization (PGO) is a compiler feature that leverages runtime profiling data to enhance the performance of Go applications by optimizing code through inlining and devirtualization. Now fully integrated into Go 1.21, PGO can be significantly enhanced using Grafana Pyroscope, an open-source continuous profiling database. Inlining involves replacing function calls with their bodies to eliminate overhead, while devirtualization optimizes interface calls into direct function calls, both improving execution efficiency based on real usage patterns. To utilize PGO with Pyroscope, developers can build an initial Go binary with profiling enabled, collect runtime profiles, query for optimal PGO data using Pyroscope's profilecli tool, and apply the PGO file during the build process. Real-world applications have shown performance improvements of 2-7% using PGO, with the potential for further gains as future versions of Go incorporate additional optimizations. This process can be implemented with Grafana Pyroscope or Grafana Cloud Profiles, providing valuable performance enhancements with minimal downsides.

Grafana Labs has released several updates, including Grafana 10.4, to address the medium severity security vulnerability CVE-2024-1442, which affects Grafana's access control system. This vulnerability, concerning data source permission escalation, allows users with permission to create data sources to potentially gain unauthorized access to all data sources within their organization by exploiting a lack of UID validation. The issue impacts specific versions of Grafana Enterprise and Grafana Cloud. Users are advised to upgrade to patched versions and ensure that only trusted users have permission to create data sources. Grafana Labs has coordinated with cloud providers like Amazon and Azure to secure their offerings and has provided detailed timelines and solutions for mitigation. They emphasize responsible disclosure of vulnerabilities and offer channels for reporting security issues, maintaining an updated security blog for public announcements.

Grafana Mimir, an open-source and scalable time-series database, faced challenges with uneven load distribution among its ingesters due to the consistent hashing algorithm previously used, leading to some ingesters being overwhelmed while others were underutilized. To address this, a new "spread-minimizing token generation strategy" was developed, ensuring a more balanced load across ingesters by optimizing token distribution within the hash ring. This strategy significantly improved load balancing in Grafana Cloud Metrics production clusters, reducing the load distribution spread from 15%-25% to 0.5%-1% and enabling better resource utilization and cost savings. The strategy, now default for clusters without shuffle sharding, involves assigning fresh tokens in a way that minimizes the difference in token ownership among ingesters, thereby achieving a more uniform distribution of in-memory time series. While the new strategy has not yet been applied to clusters with shuffle sharding due to additional complexities, it demonstrates potential for improving load balancing when coupled with enhancements in shard creation algorithms. The implementation is available in the dskit OSS Golang library, and the strategy is recommended for users of Grafana Mimir OSS or Grafana Enterprise Metrics in similar environments.

GrafanaCON 2024, scheduled to take place in Amsterdam from April 9-10, is an eagerly anticipated annual conference dedicated to the Grafana open-source ecosystem, marking its first in-person event in five years. The agenda features a mix of technical talks, live demonstrations, and user success stories, highlighting innovative applications of Grafana, such as monitoring space missions and enhancing digital preservation of Dutch literature. Key sessions from the Grafana Labs team will unveil new developments in Grafana 11 and Grafana Loki 3.0, alongside insights into best practices for observability strategies. Attendees can also look forward to workshops, community-led discussions, and opportunities to engage with project maintainers during office hours. The event will conclude with the Golden Grot Awards, celebrating outstanding dashboard creations across various categories. Despite being sold out, interested individuals can join a waitlist or access a keynote recap webinar.

Grafana 10.4 introduces a range of updates, including significant improvements to visualization tools like canvas, geomap, and table panels, along with enhanced alerting features. The release provides a preview of capabilities expected in Grafana 11 and includes a new SurrealDB plugin for integrating real-time data queries. Simplified alert notification routing and easier OAuth configuration enhance user experience and control over notifications and authentication processes. Users are encouraged to transition from deprecated AngularJS dependencies to React-based solutions to avoid disruptions. Grafana Labs expresses gratitude to its community for their contributions and encourages engagement through forums, while promoting the Grafana Cloud platform as a convenient way to access the latest features with a free tier option.

Grafana has announced an experimental integration with SurrealDB, an open-source, multi-modal database with features such as real-time collaboration, SQL and GraphQL support, and granular access control. This integration allows users to visualize SurrealDB data in Grafana, although it currently lacks some advanced features like macros and a Query Builder UI. The plugin is released in an experimental state to gather user feedback and assess its utility within the community, with plans for further development based on this input. Users are encouraged to engage through GitHub by providing feedback, reporting bugs, or contributing to the code, with the potential for a public preview and general availability release in the future. Grafana offers a forever-free tier for users to explore its capabilities in metrics, logs, and dashboards, alongside this new data source integration.

Grafana, a versatile data visualization tool, enables users to make multi-step API calls using the Infinity data source plugin, which is now officially managed by Grafana Labs. This process is particularly useful for scenarios requiring an authentication token, as demonstrated with The Cat API, where an initial request fetches a cat breed and a subsequent request retrieves an image of that breed. Although Grafana lacks native support for sequential data requests with data transfer between them, users can simulate this functionality with dashboard variables. Additionally, Grafana Synthetic Monitoring offers MultiHTTP checks that allow multiple HTTP requests and data transfer between them within a single screen, enhancing the capability for complex data interactions. The Infinity plugin’s recent adoption by Grafana Labs suggests further developments, and users are encouraged to explore the plugin's documentation and GitHub repository for more information.

Grafana Labs holds internal Hackathons three times a year to foster innovation, allowing employees to pause regular work and explore new technologies, resulting in significant contributions to the company's product development. Approximately 50% of projects from the past eight Hackathons have been implemented or are in progress, influencing the company's roadmap. Notable projects include Adaptive Metrics, which helps reduce observability costs by aggregating unused metrics, Distributed Tracing in Grafana k6 for improved system visibility during load testing, and Structured Metadata in Grafana Loki, enhancing query performance by managing high cardinality data. These innovations demonstrate Grafana Labs' commitment to a culture of creativity and continuous improvement.