November 2022 Summaries

Version 0.12.0 of the Synthetic Monitoring Agent has been released to address a high-severity vulnerability, CVE-2022-46156, which pertains to the exposure of API tokens through a debug endpoint initially enabled by default. The vulnerability, reported by a Grafana community member, has a CVSS score of 7.2, indicating significant risk for users running the agent on their local networks. Grafana Labs has responded by disabling the debug endpoint in version 0.11.2 and making further security improvements in version 0.12.0, such as allowing the use of environment variables for API tokens and setting the HTTP server to listen on localhost by default. While Grafana Labs' managed agents are not affected, users are urged to upgrade to version 0.12.0 promptly and rotate their agent tokens, reviewing their configurations to ensure secure settings. Additionally, users of previous versions should consider altering their settings to minimize exposure by configuring the HTTP listening address. Grafana encourages users to report any security issues to their dedicated email address and provides resources for further inquiries about the advisory.

On November 29, 2022, Grafana Labs released Grafana 9.3.0 and a security patch for Grafana 9.2.7 to address a high-severity stored XSS vulnerability identified as CVE-2022-31097, which was initially patched in July 2022 but reintroduced due to a build process failure. This vulnerability allows attackers to escalate privileges from Editor to Admin by deceiving an authenticated admin into clicking a malicious link. The impacted versions range from 9.1.0-beta1 to 9.3.0-beta1, and users are advised to upgrade their Grafana instances or disable Grafana Alerting as a temporary measure. The release also highlighted process improvements to prevent future regressions, including a new automated build and release pipeline. Grafana Labs remains committed to transparency and collaboration with the security research community, encouraging the reporting of security vulnerabilities via encrypted communication.

Banco Itaú, the largest bank in Latin America, manages its extensive digital operations by tracking 1.5 billion daily metrics using a combination of on-premises data centers and AWS, facilitated by Grafana and other observability tools. With 16,000 technology employees and 15,000 engineers, the bank has built an observability platform as a service to enable rapid response to issues, making use of Prometheus, Thanos, Splunk, Yaeger, and AppDynamics for monitoring and performance analysis. This system, which includes over 500 Grafana organizations and approximately 4,500 dashboards, allows for comprehensive data visualization and incident response. As Banco Itaú transitions half of its infrastructure to the cloud, they are using Grafana to monitor digital channels and ensure seamless customer experiences, while also planning to adopt SLIs and SLOs and develop a unified monitoring interface. The bank's proactive approach in observability and collaboration with AWS aims to mitigate incidents, improve resilience, and support evolving customer needs in an increasingly digital banking environment.

Grafana, an open-source data visualization tool that started in December 2013 by Torkel Ödegaard, has reached a significant milestone with over one million active instances worldwide. Initially created to make time series data more accessible and interactive, Grafana has been used in diverse applications beyond IT operations, from monitoring smart city IoT devices to helping NASA with launches. Grafana Labs CEO Raj Dutt highlighted the wide array of innovative problems the tool addresses and the various data sources it supports. With its latest release, Grafana 9, the company has improved user experience and enhanced observability and data visualization features, aiming to cater to a broad user base, from individuals to large corporations. Grafana Labs expresses gratitude to its thriving community of users, contributors, and customers, emphasizing continuous development to meet their evolving needs.

Grafana Cloud has introduced Cloud Access Policies to address the limitations of the previously used API keys, which provided only coarse-grained access controls through roles like Viewer, Editor, and Admin. The new Cloud Access Policies offer enhanced security and flexibility by allowing users to define explicit fine-grained scopes, such as metrics:read and logs:write, and realms that specify access permissions for specific Grafana Cloud organizations or stacks. This system reduces the risk of unauthorized access and supports the principle of least privilege while simplifying the management of user permissions. Additional features include filtering by labels for more refined access control, the ability to create multiple tokens per access policy, and custom token expiration options. These improvements aim to replace API keys entirely, offering better security and usability for Grafana Cloud users.

Medallia, a California-based company focused on capturing feedback signals to enhance customer experiences, has successfully unified its observability stack using Grafana. Facing challenges with divergent systems about a decade ago, Medallia's Performance and Observability Engineering team embarked on a six-year journey involving trial and error to centralize their observability tools. Initially relying on a cloud metrics platform, they transitioned to an internal system using the TICK stack and later incorporated Grafana as the visualization layer due to its support for multiple data sources. This transition helped Medallia address issues related to costs and expandability, eventually adopting Thanos to enable long-term data retention and improve querying capabilities. With Grafana as the central hub, Medallia can now efficiently manage 250 million active time series metrics across 21 environments and deliver significant insights to engineering leadership, enhancing decision-making quality at higher organizational levels. The company continues to explore integrating additional open-source tools like OpenTelemetry, Grafana Mimir, and Grafana Tempo to further optimize their observability stack.

Grafana Labs will be participating in AWS re:Invent 2022, held from November 28 to December 2 in Las Vegas, where over 50,000 attendees are expected for a variety of activities such as boot camps, labs, and keynotes. Located at Booth 745 in the Expo at the Venetian Hotel, Grafana Labs will be showcasing their open observability platform and sharing insights with visitors. Key members of their leadership team, including CEO Raj Dutt, COO Douglas Hanna, and VP of Product Tom Wilkie, will be present throughout the event. Attendees have the opportunity to meet Grafana's experts, including those specializing in the LGTM (Loki-Grafana-Tempo-Mimir) stack, either spontaneously or by scheduling a meeting in advance.

Adform, a leading advertising tech platform, transitioned from a fragmented system to a centralized observability system using Grafana to support its growth and manage more than 1,300 data sources for over 25,000 clients. Initially operating in startup mode, Adform’s disjointed tech infrastructure necessitated a move to a more cohesive system, leading to the adoption of Grafana paired with Prometheus. This change allowed Adform's DevOps team to maintain oversight while granting development teams the flexibility to choose their tools, enhancing both control and collaboration. Grafana's organizations functionality enabled separate yet visible monitoring experiences for internal teams, contributing to improved communication and cross-collaboration. The system’s transparency has significantly eased troubleshooting and reduced CPU core usage by 50%. Adform is further expanding its capabilities with successful trials of Grafana Loki and plans to experiment with Grafana Mimir, reinforcing Grafana's central role in its infrastructure.

Grafana Labs, in partnership with Isovalent, has introduced the Cilium Enterprise integration in Grafana Cloud to enhance Kubernetes network monitoring. This integration utilizes Cilium, an open-source project powered by eBPF, to provide advanced networking, security, and observability features for cloud-native environments. Designed for enterprise use, the integration allows data from Cilium Enterprise deployments to be visualized and analyzed in Grafana Cloud, benefiting system administrators, DevOps teams, networking engineers, and SREs. It includes several prebuilt dashboards that offer insights into cluster status, network flows, and security policies, alongside 17 tailored alerting rules to monitor critical components and ensure optimal performance. The integration improves connectivity observability, simplifying the monitoring and troubleshooting of complex API-driven services within Kubernetes environments.

Grafana 9.2 introduces an enhanced Grafana Loki query variable editor that simplifies the creation of dynamic and interactive dashboards by allowing users to select variables from a drop-down menu rather than hard-coding specific elements. This improvement streamlines the process of querying label names or values, accommodating both new and existing variables with backward compatibility. Additionally, Grafana 9.2 offers broader enhancements such as support for Google Analytics 4 properties, improved access control, security fixes, and more, all aimed at enhancing the user experience in creating and sharing dashboards and alerts. The release underscores Grafana's commitment to providing accessible and versatile tools, with the Grafana Cloud offering a generous free tier and various subscription plans.

In November 2022, Grafana Labs announced the launch of Grafana Phlare, an open-source continuous profiling system that became part of its observability suite, complementing metrics, logs, and traces. Following Grafana Labs' acquisition of Pyroscope in March 2023, the Pyroscope and Grafana Phlare projects merged under the new name Grafana Pyroscope. Grafana Phlare is designed to be horizontally scalable, highly available, and multi-tenant, and it helps users understand resource usage to optimize application performance and cost. The tutorial video by Grafana Labs Principal Engineer Cyril Tovena guides users through starting with Phlare, highlighting its integration with Grafana, and demonstrating its visualization features, such as Flame graphs and Top tables, which allow users to analyze CPU usage and identify performance bottlenecks. Phlare profiles are stored for long-term retention and can be queried via Grafana, utilizing the same service discovery as Prometheus, and allowing seamless correlation with other observability data. The setup process involves using docker-compose to run services for visualizing the data, and the profiling language aligns with Prometheus metrics, enabling easy navigation for users familiar with Grafana's ecosystem.

Grafana Labs conducted the Observability Survey 2022 to gather insights on observability practices and trends within their community, revealing that teams often use multiple data sources and technologies, with 67% of respondents using four or more data sources in Grafana and over half employing at least six observability technologies. Centralized observability is highlighted as beneficial, with 88% of respondents who use a single tool reporting time or cost savings, and many noting improvements in diagnostics and mean time to restore (MTTR). The survey also shows that 71% of organizations are using or considering service level objectives (SLOs) and service level indicators (SLIs), reflecting a growing interest in structured reliability metrics. The results reinforce Grafana Labs' "big tent" philosophy, which supports diverse technology integration, and respondents express excitement about learning new tools and achieving centralized observability, with popular technologies including Grafana, Prometheus, Grafana Loki, and Elastic/ELK. The survey findings were shared in the context of ObservabilityCON, where Grafana Labs announced new projects and provided insights into best practices in the observability space.

The release of Grafana Agent v0.29.0 introduces significant enhancements, notably the Grafana Agent Flow, a dynamic configuration runtime built on components, which simplifies running and configuring the agent. Key to this update is the integration of OpenTelemetry (OTel) Collector components and converters for handling traces, metrics, and logs, aligning with Grafana Labs' big tent philosophy to streamline telemetry signal management from diverse sources. This release includes 11 new Flow components that facilitate easy creation of new functionalities for converting, exporting, and receiving signals, allowing developers to use existing components as templates. These components enable the reception and conversion of signals between different formats, such as converting OTel metrics to Prometheus format, thus leveraging the capabilities of both Prometheus and OTel. The future direction of Grafana Agent, as highlighted, is centered around Flow, which is transitioning from experimental to beta in v0.30.0 and aims to serve as a broker and transformer of observability data, encouraging community involvement for further development.

Grafana Labs has released new versions 9.2.4 and 8.5.15 to address critical and moderate security vulnerabilities identified as CVE-2022-39328, CVE-2022-39307, and CVE-2022-39306. These updates resolve issues such as privilege escalation due to a race condition in HTTP context creation, vulnerability from using invitation links to sign up with arbitrary usernames or email addresses, and a username enumeration risk during password reset processes. Users are urged to upgrade their Grafana installations to protect against these vulnerabilities, with appropriate patches already applied to Grafana Cloud and coordinated with cloud providers like Amazon and Azure. Grafana Labs encourages users to report any security vulnerabilities via their dedicated security email and offers security announcements and updates through their blog and RSS feed.

The updated Windows integration for Grafana Cloud now includes support for monitoring Windows logs alongside metrics, thanks to the event log scraping capabilities provided by the embedded promtail module in the Grafana Agent. Users can easily set up this integration in Grafana Cloud by installing the Grafana Agent on their Windows machines, which allows them to send metrics and logs to their Grafana Cloud instance. Once installed, the integration provides two prebuilt dashboards: one for monitoring Windows metrics, such as system uptime, CPU, memory, disk usage, and network statistics, and another for aggregating and viewing Windows log events, including errors, warnings, and their sources. These dashboards enable users to filter data by job name, hostname, and log details, offering a comprehensive view of their Windows system's performance and event logs. This integration is available to all Grafana Cloud users, including those using the free tier, and further information can be found in the Windows integration documentation or by engaging with the Grafana Labs Community.

Grafana Labs has introduced its Writers' Toolkit to enhance the quality and consistency of technical documentation, encouraging contributions from both internal teams and global open-source contributors. This toolkit aims to streamline the documentation process by providing writing guidance, templates, and a style guide based on Google's developer documentation style guide, tailored with Grafana Labs' unique voice and tone. The toolkit is designed to reduce friction in documentation efforts, promote collaboration between developers and writers, and support scalability by aligning with technical writing best practices. It includes components that help contributors create clear, user-focused, and accessible content, organized into concepts, tasks, and references, ensuring comprehensive and accurate documentation. The initiative reflects Grafana Labs' commitment to maintaining high-quality documentation and its open-source ethos, with positive internal feedback leading to inclusion in onboarding and ongoing training efforts.

Grafana Labs has introduced k6 x Tempo, a native integration within Grafana Cloud that combines Grafana k6 load testing with Grafana Tempo tracing to enhance the analysis of performance test results and improve application reliability. This integration addresses the challenges of performance testing by bridging the gap between the external test data and internal system data, allowing for more effective root cause analysis. By correlating k6 test run data with server-side tracing data, users can better understand service behaviors during tests and generate real-time metrics for anomaly detection. Initially launched as a private beta, the integration aims to provide actionable insights to prevent reliability issues from affecting end users.

Grafana Labs has introduced Grafana Phlare, an open-source database designed for continuous profiling at a large scale, which integrates seamlessly with Grafana's existing tools like metrics, logs, and traces. This new backend aims to provide scalable, highly available storage and querying of profiling data, offering features such as native multi-tenancy and integration with object storage solutions like Amazon S3 and Google Cloud Storage. By visualizing profiling data alongside other data sources in Grafana, users can gain a comprehensive view of their systems, enhancing application performance and optimizing infrastructure costs. The release follows Grafana Labs' acquisition of Pyroscope and aims to address the challenges of continuous profiling in distributed, cloud-native environments, positioning it as the fourth pillar of observability. Grafana Phlare's ease of installation and its compatibility with various storage solutions make it a versatile tool for teams across different business units. The project reflects Grafana Labs' commitment to open source and community collaboration, with ongoing developments to enhance query capabilities and support profiling without application instrumentation.

OpenSSL vulnerabilities CVE-2022-3786 and CVE-2022-3602 have been assessed as high-impact, prompting Grafana Labs to evaluate their impact on their projects and products. The majority of Grafana Labs' core software, written in Go, is unaffected due to its reliance on Go's built-in TLS implementation, which is independent of OpenSSL. Grafana Cloud is also secure, as it depends on non-impacted or patched SSL/TLS implementations provided by cloud providers. However, some containerized releases may include vulnerable OpenSSL versions but are not confirmed to be susceptible to remote code execution. Grafana Labs is preparing updates to patch these dependencies as upstream patches become available. Security vulnerabilities can be reported to Grafana Labs via a dedicated email, and the company provides security announcements and updates through their blog and RSS feed.

ObservabilityCON 2022, hosted by Grafana Labs, highlighted a series of innovative updates and partnerships aimed at enhancing the capabilities of their observability stack. Key announcements included the acquisition and integration of Pyroscope into Grafana Pyroscope for continuous profiling, and the introduction of Grafana Faro for frontend application observability, which aims to improve the monitoring of real user interactions on websites. The event also featured the release of Grafana Phlare, a scalable backend for profiling data, and the strategic partnership with Isovalent for Cilium integration, facilitating enhanced monitoring of Kubernetes applications. Updates to the Grafana LGTM stack were unveiled, such as improved log storage in Grafana Loki, the new TraceQL query language in Grafana Tempo, and expanded metric ingestion in Grafana Mimir. Additionally, the Grafana Cloud Incident Response and Management suite saw enhancements, with new features in Grafana Incident, OnCall, and Alerting, aimed at streamlining incident workflows. These developments emphasize Grafana Labs' commitment to open source and interoperability within the observability ecosystem.

Grafana Faro is a newly announced open-source project focused on enhancing frontend application observability by introducing a configurable web SDK to capture observability signals, which can be integrated with backend and infrastructure data for comprehensive full-stack visibility. Aimed at addressing the complexity of modern browser frontends, Faro offers a JavaScript library that provides automatic instrumentation to capture logs, errors, and performance metrics, along with an event API to track user interactions. It integrates seamlessly with Grafana Cloud, allowing users to collect, process, and analyze frontend data without worrying about infrastructure management. The project is currently in a private beta phase, offering Grafana Cloud users the opportunity to trial its Frontend Application Observability service, designed to simplify real user monitoring.

Grafana Labs CEO and Co-founder Raj Dutt discussed the increasing importance of observability for companies aiming to enhance customer satisfaction during an interview on "NYSE Floor Talk" ahead of ObservabilityCON. He emphasized the necessity for businesses to ensure their software and infrastructure are functioning optimally to maintain customer loyalty. Dutt highlighted Grafana Labs' "big tent" philosophy, which promotes interoperability and allows customers to use multiple vendors and technologies without requiring data consolidation. The Grafana open-source project, founded in 2014, has experienced significant growth, with nearly a million installations globally and a community that far exceeds the customer base. ObservabilityCON, held in person in New York for the first time since the pandemic, features customers like JPMorgan and Adobe sharing their experiences, and includes the launch of two new open-source projects, showcasing the company's continuous innovation.