May 2022 Summaries

Grafana Labs has leveraged its own open-source visualization tool, Grafana, as a business intelligence solution to optimize its data stack and enhance scalability for its small team. Initially renowned for observability, Grafana has evolved to support business metrics visualization, integrating seamlessly with the modern data stack including tools like BigQuery, dbt, and Prefect. This approach allows Grafana Labs to implement a "single pane of glass" methodology, enabling interactive dashboards that merge business and observability data, while adhering to a "big tent" philosophy that supports diverse data sources. Grafana's integration with BigQuery facilitates data visualization using SQL, maintaining a clear separation between data modeling and visualization roles. The platform's features for data governance, such as user permission management and dashboard tags, further enhance data discovery and collaboration. A demo dashboard utilizing public data on Citi Bike usage in New York City exemplifies Grafana's capabilities, showcasing its interactive features and supporting user engagement through open-source community involvement.

Grafana Labs has announced the integration of Grafana Enterprise Logs (GEL) with Red Hat OpenShift 4.x, providing users with a robust solution for enterprise log management. This integration allows companies to leverage GEL's unique log indexing, storage, and administrative capabilities within OpenShift, a leading on-premises and cloud Kubernetes platform. Powered by the Grafana Loki open-source project, GEL facilitates cloud-native log aggregation with the use of LogQL, a versatile log query language, enabling efficient issue resolution and metric analysis. The deployment process involves creating a project in OpenShift, adding the Grafana Helm charts repository, and deploying the GEL Helm Chart, while ensuring compatibility by using version 2.0.4 or above. For production environments, additional configuration adjustments are recommended, such as enabling memcache for improved performance. This partnership enhances the flexibility and scalability of GEL across various cloud environments, offering enterprise-level support and configuration options.

The text discusses how to monitor an Umbrel server, which runs a Bitcoin node, using Grafana Cloud. It highlights the distinctions between fiat currency and digital currencies like Bitcoin, emphasizing the decentralized nature of cryptocurrencies with server nodes distributed across a peer-to-peer network. The guide provides a step-by-step tutorial on setting up monitoring for an Umbrel server's storage, memory, and temperature metrics using tools such as Graphite, Python, and a Raspberry Pi running Debian. By following the instructions, users can send these metrics to Grafana Cloud's Graphite service and explore them via custom dashboards, enhancing the reliability and availability of their Bitcoin node.

In an effort to address the growing demand for scalable data storage solutions, Grafana Labs conducted extensive load testing on Grafana Mimir, an open-source time series database, and Grafana Enterprise Metrics (GEM) to demonstrate their ability to handle up to 500 million active series on customer infrastructure. The initiative involved partnering with a customer who required a sustained load test at this scale, which had not been previously achieved by other vendors. By using an AWS environment that mirrored the customer's setup, Grafana Labs incrementally scaled the load, starting from 50 million active series and advancing to 500 million, while addressing challenges such as authentication bugs and AWS quota limits. The testing process involved optimizing query sharding, adjusting ingester configurations, and enhancing performance efficiencies, which resulted in significant infrastructure cost savings. Throughout the testing phases, Grafana Labs showcased the capability of GEM to manage high cardinality queries and rapid data ingestion rates, ultimately proving GEM's scalability and reliability for large-scale metric management in on-premises environments.

The updated MySQL integration for Grafana Cloud introduces a pre-built MySQL logs dashboard and enhanced Grafana Agent configuration to collect and visualize MySQL server logs, facilitating deeper insights into server performance. While metrics are commonly used for quick performance assessments, logs provide detailed insights into the root causes of issues. The integration enhances the ability to correlate metrics, logs, and traces, using the Grafana Agent to send telemetry data to the Grafana Cloud stack. The new configuration ensures synchronized labels for logs and metrics, allowing seamless navigation from metrics to corresponding logs for detailed analysis. Accompanied by a comprehensive MySQL logs dashboard, the integration retains the context of the MySQL instance and time interval across dashboards, enhancing user experience. Grafana Cloud offers a free tier and various plans, encouraging users to explore its capabilities for full-stack observability.

Grafana Cloud has introduced a new integration for macOS, allowing users to monitor their Mac systems by collecting and visualizing metrics and logs through the platform's full-stack observability features. This integration is facilitated by the Grafana Agent, which now supports macOS alongside other platforms like Linux, Windows, and Kubernetes, enabling the collection of Mac-specific data such as memory metrics. Users can easily set up the agent using Homebrew or by following alternative instructions provided in the Integrations and Connections Wizard. The pre-built macOS dashboard in Grafana Cloud displays critical metrics such as CPU, memory, network, and disk usage, and includes panels for accessing logs, which can be filtered by sender to refine search results. This integration is available to all Grafana Cloud users, with a free tier and various plans offered for different use cases, and users are encouraged to explore the macOS integration documentation and engage with the community for further assistance and feedback.

Grafana Enterprise has released versions 8.5.3 and 7.5.16 to address a moderate severity security vulnerability, identified as CVE-2022-29170, which affects versions from 7.4.0-beta1 to 8.5.2. This vulnerability involves the bypass of request security configurations via HTTP redirects from a malicious data source, potentially allowing access to forbidden hosts. The issue was discovered during an internal audit on May 2, 2022, leading to a coordinated private and public patch release with cloud providers like Amazon Managed Grafana and Azure Managed Grafana. Grafana OSS and Grafana Cloud are not impacted. Users are advised to upgrade affected versions promptly, and any security vulnerabilities should be reported securely to Grafana Labs.

The article explores the integration of Grafana, Databricks, and the Sqlyze Enterprise plugin to monitor COVID-19 virus levels in wastewater, facilitating data visualization and analysis for public health insights. The Sqlyze plugin allows Grafana to connect to a variety of SQL and non-SQL data sources using SQL queries, leveraging ODBC drivers for compatibility. By importing wastewater surveillance data from the CDC into Databricks, the setup enables the creation of a dynamic dashboard to track virus levels over time at specific treatment plants, offering visual insights into the percentage change of virus concentrations. Additionally, the dashboard supports query customization by state and county, allowing users to analyze virus levels across different regions, with the flexibility to set alerts for significant changes. The integration not only highlights the potential of using data analytics to inform public health decisions but also underscores the versatility of the Sqlyze plugin in connecting to diverse data sources.

An upgraded version of the Linux Node integration for Grafana Cloud now facilitates the correlation of logs with existing metrics, enhancing troubleshooting capabilities by allowing users to identify root causes of issues more effectively. This upgrade includes a new pre-configured dashboard based on the USE method, which emphasizes resource utilization, saturation, and errors, and introduces the ability to navigate specific timeframes where metrics display irregular behavior, correlating them with logs from various files such as syslog, auth.log, and kern.log. Additionally, the option to use systemd journal scraping enables filtering of application logs by specific units, providing a more granular analysis of issues such as disk usage spikes. The integration aims to streamline the monitoring process with features like 16 pre-built alerts and encourages users to configure their Grafana Agent for log file scraping to maximize the benefits of these enhancements.

In the podcast episode "All Things Logs," Ed Welch, Grafana Loki tech lead, and Mat Ryer discuss best practices for logging, emphasizing the importance of context in log lines to aid in troubleshooting and future accessibility. They highlight the significance of structured logging, which involves logging in machine-readable formats like JSON, and advocate for the inclusion of unique identifiers such as traceId or orderId to enhance log utility. The conversation also covers the challenges of logging at scale, suggesting Grafana Loki as a solution due to its efficient handling of log indexing through small key-value pairs, which helps manage resource costs while maintaining searchability. They stress the importance of choosing appropriate labels in Loki to avoid excessive index sizes and ensure logs are easy to locate, with examples like environment, cluster, and application being particularly useful.

Over the past year, the Grafana Observability team underwent significant restructuring to address challenges posed by its rapid growth and expanded responsibilities. This process involved splitting a larger team into three specialized squads—Observability Metrics, Observability Logs & Traces, and Observability Experience—to enhance focus, engagement, and autonomy. The restructuring aimed to improve job satisfaction, performance, and collaboration while maintaining high-quality project outcomes and community engagement. The approach emphasized team-driven solutions, which included defining problems and goals via a design document, utilizing a virtual whiteboard for brainstorming, exploring multiple proposals, and voting on the final structure. The successful implementation led to improved team dynamics, clearer career progression, and accelerated project delivery, with 100% of team members expressing satisfaction with the new structure. This restructuring highlights the importance of involving team members in decision-making processes to foster agility and long-term growth.

LogQL, the query language for Grafana Loki, offers several advanced features that enhance log querying and analysis capabilities. Inspired by PromQL, LogQL has evolved since 2020 to include extensions for extracting, transforming, and filtering logs. Notable features include the ability to count sample values per second using the unwrapped rate, calculate the length of extracted values with the text template engine, and utilize a new variable called __line__ for outputting actual log lines. LogQL allows for multiple parsers within a single query, grouping with range vector aggregations, and filtering using bytes and duration label filters. Users can also reformat log lines for improved readability, apply IP filtering, and even add emojis to logs using UTF-8 characters. Additionally, LogQL enables access to array values in JSON and supports commenting within queries, making it a versatile tool for managing logs in Grafana Loki.

Grafana Tempo and Grafana Agent are utilized for trace sampling, allowing users to efficiently manage and analyze traces by implementing tailored sampling strategies based on specific requirements. Trace sampling is vital for handling the vast number of traces generated in large-scale applications, as it helps reduce storage and processing costs without sacrificing essential insights. Grafana offers solutions like Grafana Tempo, Grafana Cloud Traces, and Grafana Enterprise Traces, which enable users to store the majority of traces with minimal sampling, allowing for both real-time observation and historical comparison. Sampling can be performed through head or tail sampling, with tail sampling being more resource-intensive but offering greater flexibility in filtering traces based on conditions like errors or specific endpoint requests. Grafana Agent supports advanced tail sampling configurations, adhering to OpenTelemetry standards, to ensure only relevant traces are retained. This capability is crucial for optimizing system performance and maintaining cost-efficiency in trace storage and analysis.

The process of collecting Prometheus metrics using the OpenTelemetry Collector and Grafana involves configuring the OpenTelemetry Collector to act as a proxy that receives, processes, and exports telemetry data in various formats, including Prometheus. The setup involves installing Node Exporter on a server to expose metrics, creating a Grafana Cloud account to store and visualize these metrics, and configuring both the OpenTelemetry Collector and Prometheus to scrape and send data to Grafana Cloud. The OpenTelemetry Collector's pipeline consists of receivers, processors, and exporters, which collect telemetry data, process it, and send it to designated outputs. Once configured, the metrics can be visualized in Grafana using pre-built dashboards, providing insights into server performance. Future discussions will explore deploying the collector in Kubernetes and using Prometheus components more efficiently.

Grafana 8.5 introduces a new feature that allows users to seamlessly transition from traces to Splunk logs, expanding the previous integration that only included Loki logs. This development enables users to connect trace data from systems like Tempo, Zipkin, or Jaeger directly to third-party logging data in Splunk, offering deeper insights into why and when incidents occur. The feature allows filtering logs by trace ID, span ID, and other metadata, making it easier to analyze extensive log data within Grafana. This enhancement was highly requested by customers who use Splunk, providing a more comprehensive approach to observability. Users can enable this feature by configuring their Splunk data source in Grafana, and the integration facilitates faster query results by offsetting the query time by one second, although users may need to adjust time ranges for more precise data. The update reflects Grafana's commitment to enhancing its observability tools, and more information can be found in their documentation or by trying the Grafana Cloud Pro trial.

Royal IHC, a prominent player in Dutch maritime history, utilizes Grafana Cloud to enhance the monitoring of their dredging vessels, which are vital for creating ports, maintaining waterways, and addressing erosion through dredging operations. Each vessel is equipped with hundreds of sensors generating significant amounts of data related to performance and environmental conditions, which are crucial for ensuring projects stay on schedule and within budget. Initially exploring various tools, Royal IHC's project leader Guus Derksen and his team found the open-source Grafana software offered the desired flexibility for integrating diverse data sources and creating custom visualizations. This allowed them to develop comprehensive dashboards that display vital metrics such as fuel consumption, production, and environmental conditions in real-time, enabling clients to make swift, informed decisions. The transition to the Grafana Cloud Advanced plan further enhanced the system's security and accessibility, allowing for secure client access and improved data management. This modern solution has significantly surpassed legacy systems, providing Royal IHC with a competitive edge in vessel monitoring.

The newly launched official ClickHouse plugin for Grafana, developed collaboratively by Grafana and ClickHouse, enhances data visualization by integrating ClickHouse's high-speed query processing with Grafana's versatile dashboard capabilities. The plugin, an open-source project on GitHub implemented in TypeScript and Go, adheres to Grafana's standards, making it lightweight and easy to maintain. It introduces ClickHouse as a data source in Grafana, allowing users to create visualizations using SQL queries or a visual Query Builder mode, and supports advanced features like Grafana macros for dynamic behavior. The plugin includes pre-built dashboards for data, cluster, and query analysis, accommodating any ClickHouse instance. Technical advancements include migrating to TCP connections and updating the frontend to React, aligning with Grafana's core components. Users can learn more through tutorials, webinars, or the ClickHouse plugin documentation, and future developments will follow an open-source workflow with ongoing contributions encouraged.

The blog post outlines the process of capturing Spring Boot metrics using the OpenTelemetry Java Instrumentation Agent. Initially, it describes setting up a simple REST service application, enabling Prometheus metrics via Spring Boot's Actuator and Micrometer, and exposing them through a Prometheus endpoint. The article then introduces the OpenTelemetry collector, which processes and exports telemetry data, demonstrating how to configure it to handle metrics in Prometheus format. Subsequently, it details transitioning from Prometheus to OpenTelemetry metrics by modifying the collector's configuration and employing the OpenTelemetry Java Instrumentation Agent. The post explains the necessity of bridging Micrometer metrics with OpenTelemetry by registering the OpenTelemetryMeterRegistry as a Spring bean to ensure compatibility and completeness of metrics. The guide concludes by emphasizing the seamless integration of Spring Boot metrics with the OpenTelemetry setup and the benefits of using Grafana Cloud for comprehensive monitoring solutions.

Observability and monitoring are crucial for contact centers using Asterisk, a popular open-source IP PBX software, to ensure seamless customer interactions. Integrating Asterisk with Grafana Cloud enhances monitoring capabilities by utilizing the res_prometheus module, introduced in Asterisk version 17, which acts as an embedded Prometheus exporter. Users can start by creating a Grafana Cloud account and enabling the exporter within Asterisk, followed by installing the Asterisk integration and configuring the Grafana Agent to gather logs and metrics. The integration offers pre-built dashboards for an overview of Asterisk metrics and log details, along with pre-configured alerts such as system restarts, reloads, high scrape time, and active call counts, allowing for efficient system monitoring and management. This setup simplifies deploying and managing Asterisk instances, ensuring a robust monitoring framework that supports scalable communication infrastructures.

Grafana Mimir is an open-source, horizontally scalable, and highly performant time series database created by Grafana Labs, designed to handle massive scale in collecting and querying metrics, surpassing limitations of existing systems like Cortex and Thanos. The database features a split-and-merge compactor to overcome storage limits and employs query sharding for faster query performance, achieving up to 40 times speed improvements in some cases. The project, named after the Norse figure Mimir, aims to maintain Grafana Labs' open-source philosophy while providing commercial viability and agility, and plans include expanding support beyond Prometheus metrics to others like OpenTelemetry and Datadog. The development of Mimir involved learning from Grafana's existing enterprise products and collaboration across teams, emphasizing continual improvement and open-source contributions.

Grafana Tempo 1.4 introduces a new optional component called the metrics generator, which produces RED metrics and service graphs from traces automatically, enhancing Grafana Cloud's capabilities and available in both Grafana Enterprise Traces 1.3 and Tempo 1.4. This update, which includes bug fixes and breaking changes, allows for seamless transition from metrics to traces by writing exemplars to the storage backend, thus maintaining existing instrumentation. Additionally, Grafana has announced plans to adopt the open, columnar Parquet format for better search performance and introduced the upcoming TraceQL query language. Users are encouraged to engage with the community for further developments and can access Tempo features via Grafana Cloud, which offers a free tier with 50GB of traces.