February 2023 Summaries

Grafana has released version 9.4, introducing updates like enhanced navigation and custom visualization panels, alongside crucial security patches addressing vulnerabilities CVE-2023-0594, CVE-2023-0507, and CVE-2023-22462. These vulnerabilities involve stored XSS issues in the TraceView panel, geomap plugin, and text panel plugin, each requiring users with Editor roles to exploit. The CVSS scores indicate high to medium severity, highlighting the potential for vertical privilege escalation if malicious JavaScript is executed by users with elevated permissions. To mitigate these risks, users are advised to upgrade their Grafana instances or enable the Content-Security-Policy option. Grafana encourages reporting any further security issues via their security email, with a preference for encrypted messages using their PGP key.

Siemens Mobility, a rail transportation company, is leveraging Grafana Enterprise to transform its trains into advanced data-driven systems, enhancing both maintenance operations and customer service. By treating trains as "data centers on wheels," Siemens Mobility is employing Grafana to rapidly process millions of data points from train sensors, thereby enabling the identification of potential faults and optimization of maintenance activities. This digital transformation, led by Martin Klimmek and his team, includes upgrading older trains with smart technology and aims to establish a self-service system for staff and customers. The initiative not only supports seamless operation and proactive maintenance for Siemens' rolling stock but also represents a broader vision of integrating observability tools across all aspects of rail service, showcasing the potential for technological advancement in the transportation sector.

The XYZ chart is a new three-dimensional visualization tool introduced in Grafana, designed to reveal connections between three variables, enhancing traditional two-dimensional data visualization. Despite historical challenges in interpreting 3D charts, the XYZ chart aims to offer powerful insights with proper design and interactive features, such as rotating, zooming, and panning. It is currently in alpha development, requiring users to modify their configuration to enable it, and promises applications across diverse fields like business intelligence, academia, and geographical mapping. The chart automatically assigns X, Y, and Z fields from data but allows manual adjustments for detailed analysis, with customizable colors and sizes for data points. Future development plans include adding features like multiple series support, various data point shapes, and regression lines to increase its versatility and functionality, with feedback welcomed through Grafana's community channels.

Prometheus metrics, often visualized in Grafana, can sometimes require a shift in focus from numerical values to the labels themselves, which contain valuable information. This guide outlines how to extract and visualize Prometheus label values using Grafana's dashboard capabilities. By setting up a Prometheus data source and selecting appropriate metrics, users can create a table visualization rather than a standard graph to better display categorical label data. The process involves using PromQL queries to filter and customize the data display, removing unnecessary fields like time and value, and utilizing data transforms to further refine the information shown. This method allows for greater insights into specific metrics, such as identifying software versions, and can be adapted to various Prometheus metrics for enhanced data analysis and visualization.

Wells Fargo embarked on a significant transformation of its observability stack in late 2021, moving away from a complex and outdated system to a more streamlined and efficient setup using Grafana Enterprise and Grafana Cloud. Led by Senior Software Engineering Manager Nikhilesh Tekwani, this initiative aimed to unify observability data, automate alerts, and deliver dashboards to enhance monitoring capabilities. The transition involved overcoming challenges with legacy solutions that were not cloud-ready and required extensive manual work, as well as migrating over 25,000 alert rules to a modern system. By implementing Grafana Agent and Grafana Mimir, Wells Fargo improved data aggregation and visualization, while integrating with platforms like ServiceNow for incident management. Within 10 months, the team achieved significant milestones, including the establishment of out-of-the-box dashboards and increasing active metric consumption significantly. Despite the progress, Tekwani emphasizes that the journey is ongoing, with plans to expand the observability team and continue modernizing their systems.

Nikhilesh Tekwani, Senior Software Engineering Manager at Wells Fargo, is spearheading a transformation in the bank's observability strategy by migrating to Grafana Cloud and Grafana Enterprise. Tekwani emphasizes a management philosophy that balances work and personal life, encouraging his globally dispersed teams to plan vacations while maintaining high performance. He prioritizes regular one-on-one meetings to understand both team and individual priorities, and fosters an environment where every team member's input is valued, allowing the best ideas to prevail. Tekwani looks for candidates who are respectful, collaborative, and not afraid to express themselves, valuing expertise in niche areas of observability like metrics and logging. He uses humor to break the ice in interviews, aiming to reveal candidates' true personalities. Despite the challenges of managing a hybrid team across various locations, Tekwani focuses on enhancing communication and collaboration, utilizing tools like Microsoft Teams. He stresses the importance of a balance between empowerment and accountability, particularly in managing risks within the financial institution, and seeks team members who are committed to self-improvement and knowledge sharing.

In early 2022, Clearco, a leading e-commerce investor, faced challenges with its alerting system, where issues were often discovered by the Customer Success team before notifications were triggered. To address this, Clearco transitioned to Grafana Cloud, utilizing its Incident Response and Management suite, which includes Grafana Alerting, Grafana OnCall, and Grafana Incident, to enhance their incident management and alerting capabilities. This transition began in late summer 2022, with substantial adoption across their 40-person engineering team, as approximately half of the team now actively uses these features. The move to Grafana Cloud enabled Clearco to consolidate their metrics and alerting processes, allowing for efficient team-based alert routing and streamlined incident response, which improved clarity and reduced response times during outages. The company has received positive feedback on the new system, which facilitates a more structured and automated incident management process, reducing confusion and stress during critical situations, and providing a clear protocol for resolution.

Ed Welch, a leader at Grafana Labs, shares his experience of using a 3D printer to create a solar radiation shield for his weather station, which required a durable material called ASA that releases potentially hazardous particulates. To address air quality concerns in his non-ventilated basement, he constructed a homemade enclosure using materials from IKEA and attempted to filter the air with a bayonet-style respirator filter and computer case fans. Despite initial failures, he successfully implemented a system to monitor and control air quality using sensor data displayed on Grafana dashboards, finding that the particulates and VOC levels remained low even without the fan. This project, while possibly overkill, provided him with valuable insights and peace of mind, underscoring his passion for building and problem-solving.

Grafana Labs conducted a series of unconventional load tests using Grafana k6 to explore the performance limits of their load generation system, focusing on the system's ability to handle high volumes of data and high-cardinality metrics. Unlike typical load tests that concentrate on the system under test (SUT), the focus here was on the load generator's capacity, which involved atypical design choices like using an invincible SUT and generating unrealistically large loads. The tests—peak, spike, and soak—revealed insights into system bottlenecks, particularly in storage and database locking, and highlighted the need for better aggregation and autoscaling strategies. Grafana Labs emphasized the importance of monitoring alongside testing to understand system health and user experience, and noted that different test types can uncover distinct system weaknesses. The findings reinforced the notion that load testing must be tailored to each system's unique characteristics and that the real best practice involves thoughtful, system-specific test design.

The introduction of the Embrace data source plugin for Grafana offers mobile development teams enhanced capabilities to visualize and analyze real-time mobile app data alongside over 300 other plugins available on Grafana. This integration allows mobile engineers to share critical insights, known as golden signals, with various specialized teams, thereby eliminating data blind spots and achieving full-stack visibility into user experiences. By using Grafana's comprehensive dashboarding and visualization tools, teams can seamlessly correlate mobile metrics with existing logs and infrastructure health data, improving cross-team collaboration and issue resolution. The plugin facilitates a more thorough understanding of mobile app health and performance, allowing for accurate impact analysis and recovery confirmation, ultimately enhancing the end user experience. The Embrace plugin offers out-of-the-box dashboards for visualizing crash metrics and KPIs, and its simplicity of installation via the Grafana website enables immediate access to real-time Embrace metrics.

Grafana Faro and xk6-browser, part of the Grafana Labs open-source ecosystem, are innovative tools designed to enhance frontend observability and performance testing. Faro offers a customizable SDK for web app instrumentation, enabling the capture of observability signals that can be integrated with backend and infrastructure data. Released in November 2021, xk6-browser utilizes browser-level APIs to gather frontend performance metrics for Grafana k6 tests. These tools collectively provide insights into the health of frontend applications by addressing the importance of browser instrumentation in capturing metrics, logs, and traces that backend monitoring might miss. The episode discussed their integration, showcasing how browser-based monitoring complements performance testing, and highlighted a demo using the Webshop application to illustrate how these tools work together. Despite some challenges in aligning metrics and API maturity, the tools demonstrate significant potential in identifying browser-level errors and improving user experience. The discussion also emphasized the importance of distributed tracing and hybrid approaches to performance testing, underscoring the synergy between Grafana and k6 for comprehensive observability.

Jose Vicente Nunez, a system administrator and DevOps professional at a fintech company, attended ObservabilityCON 2022 in New York City with the hope of gaining practical insights for his work with Grafana, a tool he has used since 2020 for infrastructure monitoring. Initially skeptical about vendor events, Nunez found the conference highly valuable, especially the workshops and presentations that provided practical examples and techniques for optimizing alert systems using Grafana Mimir. The event facilitated networking with Grafana engineers and industry peers, offering opportunities for 1:1 coaching and collaboration. Particularly impactful were the sessions on scaling and monitoring multiple services, which aligned closely with Nunez's responsibilities of providing metrics to guide business and technology decisions. He appreciated the accessibility of conference materials post-event, enabling colleagues who couldn't attend to benefit from the sessions. The experience was further enriched by meeting Grafana Labs' solutions engineering manager, Ronald McCollam, and the opportunity to discuss his book. Nunez deemed the event a worthwhile endeavor and recommended it to others, expressing enthusiasm for future participation.

Flaky tests, which unpredictably succeed or fail without changes to the code, pose significant challenges in software development, particularly during continuous integration (CI) processes. At Grafana Labs, they tackled this issue by utilizing Drone CI, a container-native CI tool, alongside Grafana, Prometheus, and Grafana Loki to monitor and manage these tests. By exporting environmental variables from Drone CI and creating custom metrics with a Prometheus exporter, the team could visualize build data and set up alerts for CI failures. They also used Grafana Loki to query Drone logs, helping identify and address flaky tests by running repeated builds and analyzing success ratios. This approach allowed them to delegate troubleshooting to the right teams and improve their build success rate, establishing alerts to maintain it above a set threshold. Although the complete eradication of flaky tests is unlikely in large projects, the observability and responsive measures implemented have significantly enhanced the robustness and efficiency of their CI pipelines, ensuring timely and reliable releases of their products.

TraceQL is a new query language introduced by Grafana Labs, designed to enhance the extraction of valuable insights from distributed tracing data using Grafana Tempo, an open-source tracing backend. TraceQL enables users to effectively query and analyze traces, which are structured as trees with components like spans and attributes, by offering a syntax inspired by PromQL and LogQL. It allows for complex queries using boolean logic and supports intrinsics and customizable attributes, with robust data-type awareness for operations on text, numeric, and duration fields. TraceQL also features on-the-fly processing pipelines for aggregations, as well as scope targeting for OpenTelemetry’s resource-level and span-level attributes. Future enhancements include advanced querying capabilities for event sequences and expanded processing pipelines. Currently, TraceQL is available in Tempo 2.0, with its user experience integrated into Grafana 9.3+, providing a powerful tool for analyzing microservice environments and improving trace data utility.

GrafanaCON 2023, a major virtual event organized by Grafana Labs, will take place from June 12 to 14 and will feature workshops and sessions led by experts from the Grafana community to celebrate a decade of Grafana dashboards and the release of Grafana 10. The conference will cover various Grafana Labs open-source projects like Grafana Loki, Grafana Mimir, Prometheus, and Grafana Tempo, alongside significant updates within the Grafana ecosystem. The event is free to attend, with registration available online, and it provides opportunities for participants to connect with over 20,000 community members globally. The call for presentations is open until March 10, inviting submissions from diverse and first-time speakers to share their experiences with the Grafana LGTM stack, whether in personal, academic, or professional contexts. Participants can look forward to engaging with past sessions on demand and exploring other Grafana Labs events for further community interaction.

Grafana has unveiled a redesigned technical documentation aimed at enhancing accessibility, modernity, and scalability. The updated design features a new typeface, layout, improved navigation, and a floating table of contents for better user experience, ensuring users can find information efficiently. The redesign facilitates better access to related resources, while also increasing opportunities for community feedback, which has led to a rise in user engagement and contributions. Over the past year, the documentation team has expanded significantly, reflecting Grafana's commitment to developing comprehensive and user-friendly resources for its open-source projects and products. Future plans include further evolution of documentation to align with upcoming Grafana releases and improved resources for Grafana Cloud users, inviting community contributions through their Writers' Toolkit.

Grafana Labs leverages and contributes to the open-source project OpenCost to enhance real-time cost monitoring in Kubernetes, addressing the challenge of insufficient cost visibility in the rapidly growing adoption of Kubernetes. Developed by Kubecost and released as a CNCF sandbox project in 2022, OpenCost offers real-time insights into Kubernetes spend and resource allocation, breaking down costs across nodes, persistent volumes, and load balancers. Grafana Labs has utilized OpenCost to improve cost observability, especially in obtaining accurate vCPU and memory cost data and monitoring costs across multi-cluster architectures. Their contributions to OpenCost focus on observability improvements, ease of multi-cluster deployment, and performance enhancements. By implementing industry-standard operational metrics and utilizing prom-label-proxy for efficient data querying, Grafana Labs aims to optimize OpenCost's performance and resource utilization. Their efforts include releasing Grafana dashboards and alerting rules and planning for broader community engagement through open-source contributions and collaborations within the OpenCost community.

Grafana Tempo 2.0 introduces significant advancements with the integration of Apache Parquet as its default storage format and the introduction of TraceQL, a new query language designed for discovering traces. Inspired by LogQL and PromQL, TraceQL offers users a straightforward learning curve and provides unique capabilities not found in other open-source tracing solutions. This release aligns with Grafana's vision of creating a comprehensive distributed tracing database while allowing users the flexibility to build custom tools and insights. The shift to Parquet, though requiring more resources, enhances the accessibility and utility of trace data by storing it in an open format. Tempo 2.0 also includes several bug fixes and breaking changes from its previous version, emphasizing improved performance, configuration cleanup, and the potential for future feature expansions. As Grafana Tempo continues to evolve, users are encouraged to engage with the community through various channels and explore new features, particularly in the context of large-scale tracing operations.

Grafana Labs has released Grafana Agent v0.31, introducing significant updates including a new Helm chart for easier Kubernetes deployment, Windows Docker image support, and enhanced capabilities for continuous profiling with Grafana Phlare, following the acquisition of Pyroscope. The release allows for seamless integration with Kubernetes through the Helm Chart, enabling easy installation, while also supporting profiling data collection in Google's pprof format to be stored in a Phlare database. The update enhances the Grafana Agent's ability to convert log formats between OpenTelemetry and Grafana Loki, facilitating log management across platforms. Furthermore, the Grafana Agent now features integration with Grafana Mimir, allowing auto-discovery of Kubernetes resources using PrometheusRule CRDs, thus optimizing alerting and query processes. The company provides comprehensive documentation and community support to assist users in adopting these new features, underscoring their commitment to creating open-standard, easily installable products.