August 2022 Summaries

Callum Styan from Grafana Labs discusses a service outage in Grafana Cloud Logs caused by the incorrect use of Kubernetes label selectors during a migration process. The incident occurred while transitioning Grafana Cloud Logs clusters from a separate key-value store to an embedded memberlist to eliminate a single point of failure. An additional label selector was mistakenly added, leading to misdirected service discovery and overwhelming a portion of the pods, resulting in a 25-minute outage. The outage was resolved by scaling distributor deployments horizontally, which spread the load more evenly, and by reverting the configuration change. The investigation revealed that the issue was due to the unintentional inclusion of label selectors on services that filtered pods incorrectly, compounded by the unique scaling configurations of different environments. The team used extensive metrics and logging to identify the problem and has since updated their configurations to prevent similar issues in the future. The article serves as a learning resource for others using Kubernetes label selectors.

Grafana has released new versions, including Grafana 9.1.2 and Grafana Image Renderer 3.6.1, to address a high-severity security vulnerability identified as CVE-2022-31176, which poses a risk of unauthorized file disclosure when the Grafana Image Renderer plugin is used with HTTP remote rendering. This vulnerability, discovered during an internal security review on July 21, 2022, has a CVSS score of 8.3 and could allow malicious users to access unauthorized files under certain conditions. To mitigate the risk, users are advised to upgrade their Grafana installations and the Image Renderer plugin, configure strong security tokens, and restart their systems. The issue has been addressed in all affected versions, with patches applied to Grafana Cloud and coordination with cloud providers to ensure security. Grafana also encourages reporting any security vulnerabilities through their designated email, offering a PGP key for encryption.

Grafana OnCall, a tool for managing on-call schedules and escalation chains, aims to simplify and reduce errors in these processes, particularly when used with as-code workflows. The introduction of a Terraform provider for Grafana OnCall, available for both Grafana OnCall OSS and Grafana Cloud, facilitates this by allowing users to manage integrations, schedules, and escalations programmatically. The setup process for Grafana OnCall via Terraform involves creating API tokens, configuring the provider, and establishing integrations and escalation chains. The guide explains how to create on-call schedules using Terraform, including setting up on-call rotations and escalation chains, and routing alerts based on severity. The difference in handling default and critical alerts is highlighted, with critical alerts receiving immediate attention and default alerts having a delay to limit distractions. Future plans include a Terraform exporter to assist teams in migrating to Infrastructure as Code, and users are encouraged to explore Grafana OnCall's documentation and community resources for further support.

ObservabilityCON 2022, hosted by Grafana Labs, is transitioning from a virtual to an in-person event, taking place in New York City on November 1-2. This premier open source observability conference will feature a blend of workshops and a full day of technical presentations, interactive demos, and expert advice sessions. Attendees can expect to enhance their skills with workshops on dashboarding, gain insights from technical sessions on metrics management and updates on Grafana products, and hear success stories from the community. The event will also offer opportunities to engage with industry leaders and explore new strategies for achieving business goals through open source observability. With limited seating and early bird pricing, participants are encouraged to register early to secure their spot. Additionally, applications are open for those interested in presenting their own observability experiences and best practices at the conference.

Grafana 9.1 introduces file provisioning for Grafana Alerting, allowing users to configure their alerting stack using files on disk, similar to how data sources or dashboards are managed. This new feature enables flexible organization of alerting data, supporting various layouts like team-based or object-type files. Upon start-up, Grafana reads files in the alerting subdirectory, updating, inserting, or deleting objects as necessary, with changes only possible through file modifications and a subsequent restart or API call. In Kubernetes environments, file provisioning can be utilized through config maps, removing the need for a persistent database and ensuring resources are retained after restarts. Additionally, provisioned objects in the Grafana UI are marked as immutable, preventing manual edits. This feature enhances the Grafana Alerting experience, which began with Grafana 9.0, and is part of Grafana Cloud's broader suite of tools for monitoring metrics, logs, and dashboards.

Grafana 9.1 has introduced service accounts as generally available, enhancing machine-to-machine interactions by allowing automated or compute workloads to interact with Grafana's APIs securely and efficiently. Service accounts are linked to organizations rather than individual users, facilitating easier management and token rotation without the need to change tokens when administrators leave. They are designed to minimize permission sprawl by allowing specific roles and permissions tailoring, which reduces the risk of compromising the system. The new system also supports the migration of existing API keys to service accounts, enabling multiple tokens per account for better token management and security. Service account tokens now have a visually identifiable pattern with a "glsa" prefix and a checksum to help developers recognize and verify tokens in codebases, reducing false positives in open source secret scanners. The Grafana AuthNZ team plans to continue enhancing authentication and authorization features to bolster security in Grafana, with comprehensive documentation available for those interested in implementing service accounts.

AWS Lambda is a serverless computing service that allows users to execute code on-demand without managing underlying infrastructure, but it is often seen as a "black box" due to its lack of transparency in cost, performance, and reliability. Continuous profiling provides a solution to these challenges by analyzing the performance of Lambda functions and identifying resource consumption, enabling optimizations. Pyroscope's AWS Lambda extension offers a native profiling approach that minimizes overhead on the critical path by offloading profiling tasks to an asynchronous path, allowing users to gain insights into their functions without impacting performance. The Pyroscope UI further aids in analysis with tools like flame graphs, time controls, and diff views, facilitating function optimization and resource allocation. Integrating Pyroscope's Lambda extension into a function involves adding a layer and configuring environment variables for profiling data transmission.

Grafana Tempo 1.5 introduces several significant updates, most notably the experimental support for a Parquet-based columnar store, which enhances data retrieval efficiency by reducing the amount of data needed per query and enables future development of TraceQL, a new trace query language. The release also includes usage reporting to better understand community use, as well as extensions to the metrics generator with the inclusion of queues and databases in service graph metrics, leveraging OpenTelemetry conventions. Bug fixes address various performance issues, while breaking changes align span metric names with OTel conventions and require Jsonnet users to adjust storage configurations. Looking ahead, plans for Grafana Tempo 2.0 involve further integration of Parquet with TraceQL and rewriting WAL code to facilitate TraceQL searches. Users are encouraged to engage with the Grafana community for support and updates, with the option to explore Grafana Cloud for an easy start.

Energinet, an independent public enterprise owned by the Danish Ministry of Climate, Energy, and Utilities, uses Grafana Enterprise to monitor and manage their high-voltage substations and underwater energy cables, helping them detect and address issues promptly to prevent revenue loss. The transition from using Excel for data collection to Grafana has significantly improved Energinet’s ability to monitor real-time data from 10 to 15 sources, set up alerts for specific metrics, and conduct effective detective work when cables are damaged. Grafana's easy-to-use dashboards and alerting capabilities allow Energinet's Asset Preparedness team to pinpoint issues such as oil leakages and identify external causes like vessel anchors damaging cables, leading to successful liability claims against offenders. Since adopting Grafana in early 2022, the user base at Energinet has grown from 30 to about 120, with expectations for further growth, underscoring the tool's integral role in their operations and risk management.

Grafana 9.1 introduces the "trace to metrics" feature, enhancing the tool's ability to correlate different telemetry signals by allowing users to navigate from a trace span to a selected data source, such as Prometheus. Currently in beta, this feature enables users to view trends or aggregated data related to each span by configuring tracing data sources and linking queries with specific tags. These queries dynamically insert span attribute values, facilitating seamless navigation between traces and metrics. The feature works well with the metrics generator introduced in Tempo 1.4, which produces metrics highly correlated to traces. Grafana encourages users to explore additional use cases, provide feedback, and engage with the community to further refine and expand this functionality.

Grafana 9.1 introduces the public dashboards feature, enabling users to share live, read-only versions of their dashboards with anyone, regardless of their Grafana organization affiliation. Previously, dashboard sharing required either taking a one-time snapshot or disabling all authorization, both of which had limitations. This new feature, initially available in alpha and requiring a feature toggle or support contact for Grafana Cloud users, allows secure sharing by executing queries stored in the database on the backend, thus protecting against unauthorized access. While public dashboards might increase data source traffic, Grafana Enterprise and Cloud Pro users can utilize query caching and rate limiting to manage this. As an alpha feature, it is considered a minimum loveable product, with future enhancements and user feedback being encouraged through a public GitHub discussion. Documentation provides guidelines on enabling the feature and its limitations, while Grafana Cloud offers a free tier and various plans to accommodate diverse user needs.

The article details the journey of the author, Matt Dodson, as he learns JavaScript and load testing using Grafana k6, an application for load testing with scripts written in JavaScript. Despite having limited programming experience, Dodson was challenged by the k6 team with eight tasks to familiarize himself with both JavaScript and the k6 environment. These challenges, ranging from configuring test options to using custom metrics and modularizing scripts, served as a practical introduction to the k6 API and script writing. The author reflects on the learning process, acknowledging the intricacies and time consumption involved in acquiring new technical skills, the importance of practical experience, and the value of clear and detailed documentation. He emphasizes the necessity for iterative practice and how documentation can enhance learning by providing context and examples. Throughout the learning experience, Dodson also gained insights into the nuances of scripting, such as the importance of debugging, modularization, and understanding execution contexts.

At GrafanaCONline 2022, Theia Scientific's Chris Field and Volkov Labs' Mikhail Volkov presented on using Grafana and machine learning for real-time microscopy image analysis, a process traditionally manual and time-consuming. By integrating machine learning, computer vision, and Grafana, they automated workflows to enable real-time analysis, significantly reducing labor costs and delivering rapid results. The Theiascope concept combines edge computing, machine learning, and web technologies, allowing for effective image analysis even in environments without internet access. Grafana was chosen for its robust visualization and customization capabilities, and the project has been deployed at locations like Argonne National Laboratory, where it operates multiple models simultaneously. The platform's features include a customizable interface and real-time export options, enhancing its utility for researchers and scientists.

Grafana has introduced a new feature called usage groups in its Grafana Cloud Advanced plan, aimed at helping centralized observability teams and administrators track and attribute metrics usage back to different groups within a shared Grafana stack. This feature aids in governance and cost control by allowing metrics to be attributed to various teams, environments, or services, depending on the organization's setup. Users can choose to maintain their Grafana stacks in three ways: a single shared stack for the organization, individual stacks for each team, or separate stacks for different environments like development, staging, and production. The usage groups provide additional insights into metrics usage, allowing organizations to analyze and better understand their data consumption, which can be particularly useful for financial planning and efficient resource allocation. The feature supports configuring metrics based on services or environments, enabling organizations to track and manage their metrics effectively. Grafana Cloud offers a free tier and additional plans to cater to varying user needs, encouraging users to explore this new capability and optimize their Grafana setup.

Grafana Cloud has introduced an enhanced Docker integration for monitoring containerized applications, addressing the complexities brought by the increasing use of Docker among developers. This upgraded integration, accessible to all Grafana Cloud users including those on the free tier, features prebuilt dashboards for monitoring Docker logs and metrics with the help of the Grafana Agent. Users can easily set up the Docker integration by creating a Grafana Cloud account, installing the integration through the platform, and configuring the Grafana Agent to collect data. The integration offers two main dashboards: the Docker overview dashboard, which provides overall statistics and container utilization metrics, and the Docker logs dashboard, which offers insights into Docker log files, errors, warnings, and log sizes, all of which can be filtered and navigated through seamlessly. This setup facilitates efficient troubleshooting and monitoring of Docker instances, and further information is available through the Docker integration documentation or the Grafana Labs Community Slack channel.

MongoDB, a leading NoSQL database, is widely utilized by developers for storing various application metrics. The MongoDB Enterprise plugin for Grafana, accessible to users with a Grafana Cloud account or a Grafana Enterprise license, facilitates the visualization, exploration, and monitoring of data stored in MongoDB by integrating it with Grafana's capabilities. A straightforward video guide demonstrates how to quickly set up this integration, allowing users to configure Grafana and execute MongoDB queries without modifications. Once configured, users can create dynamic Grafana dashboards using template variables and diverse visualizations. Grafana Cloud offers a 14-day free trial with access to features like unlimited metrics and an Enterprise plugin, encouraging new users to explore its observability solutions. Additionally, an upcoming webinar will delve deeper into unifying data with Grafana plugins, further enhancing understanding and usage of these tools.

Fuelics, a company specializing in battery-operated narrowband IoT sensors, utilizes Grafana and Grafana Loki for monitoring and visualizing data in smart city environments. During their presentation at GrafanaCONline 2022, Fuelics experts Ioannis Nikolau and Marios Klavdianos explained how their NB-IoT sensors collect vital information such as fuel, water, and waste levels, transmitting this data to the cloud for comprehensive analysis. By employing Grafana Loki, Fuelics ensures efficient data parsing and storage, maintaining the correct time series order and optimizing data streams. Their use of Grafana dashboards allows for intricate visualizations of sensor data, including device metrics and event cross-checking, to ensure measurement accuracy and sensor health. This approach enables the Fuelics team to manage large-scale sensor deployments effectively and deliver precise insights to clients through a user-friendly web application.

Grafana Labs has introduced support for deploying Grafana Enterprise Metrics (GEM) on Red Hat OpenShift 4.x, an on-premises enterprise Kubernetes solution that also operates in cloud environments. GEM is a leading enterprise metrics solution powered by the Grafana Mimir open source project, offering secure, scalable, and cost-effective operations. This integration allows customers who use OpenShift, whether on-premises or in the cloud, to leverage GEM's capabilities for collecting metrics and using PromQL for queries to resolve issues and analyze data effectively. The deployment process is facilitated through a series of steps involving the GEM Helm Chart, which supports OpenShift's unique security model by allowing users to specify either Security Context Constraints (SCC) or Pod Security Policy (PSP). The setup is designed to be straightforward, requiring configuration adjustments such as enabling memcache for production use and specifying block storage options like Minio for testing. This advancement offers a flexible and efficient solution for enterprises seeking robust metrics management within their OpenShift environments.

Amid rising concerns about global air pollution and its severe health impacts, the Air Quality Project emerged as an innovative "activist engineering" initiative by DesignSpark, part of the RS Components community, aiming to educate and empower engineers to design sustainably. At GrafanaCONline 2022, Pete Wood highlighted the role of Grafana Cloud in the project's development, using its platform to visualize and manage data from the Environmental Sensor Design Kit created to measure indoor pollutants like VOCs and CO2, albeit with a focus on trends rather than precise accuracy. The kits were initially distributed to 12 engineering influencers worldwide, resulting in projects like Michael Omotosho's Mindful Droid, which raises pollution awareness in schools, and Jude Pullen's CO2 Canary, which alerts users to high CO2 levels. These efforts are supported by Grafana's customizable dashboards, which aid in data visualization and foster innovation by gamifying air quality monitoring. As the initiative progresses, DesignSpark plans to expand its network to 50 more engineers and release data publicly to inspire broader participation, acknowledging Grafana Cloud's pivotal role in enabling this activist engineering movement.

Released with Grafana 9.0, the new Grafana Alerting system is the default alerting platform across all Grafana products, featuring significant improvements, enhanced documentation, and video tutorials to ease the process of creating alerts. Users can establish three types of alert rules: Grafana-managed alerts, which handle complex conditions across multiple data sources; Mimir- or Loki-managed alerts, which use PromQL or LogQL queries executed independently from Grafana; and recording rules for precomputing expensive queries. A video tutorial demonstrates setting up a Grafana-managed alert for detecting HTTP server errors, with options for classic conditions or reduce and math expressions, enabling complex alert conditions. Grafana-managed alerts offer customizable evaluation intervals and durations, as well as features for organizing alerts into folders and groups, adding annotations, and providing additional context such as summaries and runbooks. Mimir-managed alerts, useful for high latency or costly queries, allow users to group alert rules with namespaces and add custom labels for routing. The new alerting features are default in Grafana 9, with a rollback option to legacy alerting, which will be phased out in Grafana 10. More information can be found in Grafana's documentation, GrafanaCONline 2022 sessions, and through their cloud platform offerings.