August 2021 Summaries

Debugging microservices can be a daunting task due to their complex architecture, but tools like Istio, Grafana Tempo, and Loki significantly simplify this process. These tools enhance traceability and logging, allowing developers to quickly isolate issues by using unique Trace IDs and spans to filter relevant data. Tempo and Loki, part of the Grafana Stack, seamlessly integrate with Grafana dashboards, providing an intuitive visualization of system components and interactions. This integration not only facilitates faster debugging but also improves overall efficiency by reducing the time required for postmortem analyses, thereby enhancing business productivity. The Grafana Stack's capabilities are further complemented by its integration with cloud storage solutions like S3, easing data maintenance and storage requirements. By leveraging these tools, developers can streamline their workflows, making it easier to manage and troubleshoot large-scale microservices environments.

The Honeycomb plugin for Grafana, now available for Grafana Cloud and Enterprise users, allows for integration of Honeycomb's high-cardinality and high-dimensionality data into Grafana's observability platform. This partnership enables users to query and visualize Honeycomb data within Grafana, facilitating the monitoring and debugging of application performance issues by linking directly to Honeycomb's analytical features. For instance, when a Grafana alert signals increased latency, users can identify and investigate specific issues by leveraging Honeycomb's capabilities such as viewing detailed traces and using the BubbleUp feature to discern anomalies. This integration reflects Grafana Labs' commitment to offering flexible observability solutions that empower users to manage their data across diverse systems.

Grafana Tempo 1.1 has been released, featuring notable improvements such as hedged requests that significantly reduce latency by allowing retries for slow backend requests, resulting in a 45% performance boost. The update introduces a tenant index that enhances performance and reduces the total cost of ownership by minimizing the frequency of backend queries, leading to a 60% cost reduction. New caching parameters provide operators with better control over cache space, addressing issues like excessive cache requirements for bloom filters. Bug fixes include resolving startup errors for queriers and improving ring reliability by addressing issues with unhealthy ring members using memberlist. The release also deprecates older block formats, urging users to upgrade to ensure compatibility with future versions. Tempo's release cycle remains bi-monthly, with exciting features like native search on the horizon for version 1.2. Users are encouraged to engage with the Tempo community through various platforms and explore Grafana Cloud's free tier to get started with Tempo.

Grafana Enterprise Metrics 1.5 introduces significant updates aimed at improving multi-tenancy and query management for observability teams. Key enhancements include per-tenant usage metrics, allowing operators to monitor system usage at a granular level to manage resources and costs effectively. These metrics are collected via telemetry and displayed on pre-built dashboards that facilitate easy monitoring of each tenant's activity relative to set limits. Furthermore, the release simplifies querying across multiple tenants by introducing a wildcard character, which streamlines the process of accessing data across various tenants without needing to specify each one individually. This feature enhances user experience by reducing the complexity of query formulation, although caution is advised to avoid overwhelming the system. These updates are designed to enhance the functionality and user-friendliness of GEM, especially for those offering Prometheus-as-a-service.

A webinar hosted by Grafana Labs is scheduled for August 26, focusing on the challenges and advantages of scaling Prometheus and Graphite metrics within observability strategies. Presenters Jen Villa and Ali Gerrard will discuss common issues encountered when scaling these metrics and demonstrate how Grafana's observability stack can unify metrics across different systems. The session will cover both real-time and historical analysis capabilities, applicable in Grafana Cloud or self-hosted environments, offering insights into enhancing metric management and analysis.

Grafana Labs has secured $220 million in Series C funding to further enhance its open-source observability platform and support its community and customers. This funding follows previous rounds in 2019 and 2020 and aims to accelerate platform development. Grafana Labs has introduced a free tier of Grafana Cloud, offering extensive no-cost observability solutions, including metrics, logs, and traces, with the recent addition of 50GB of traces via the Grafana Tempo OSS project. The company has expanded its platform to include Loki for logging, Tempo for tracing, and integrations with over a hundred data sources, alongside the acquisition of k6 and Pace.dev to enhance developer experience. Grafana Labs began with a single open-source project, revolutionizing data visualization with user-friendly dashboards, and remains committed to open-source investment and delivering on user needs.

Grafana Labs' acquisition of k6, an open-source load-testing tool, has significantly enhanced their development processes by allowing them to use the tool's JavaScript scripting capabilities to test the performance of Grafana's features such as the short-lived token solution. Initially chosen for its compatibility with Grafana's Go-based backend and JavaScript frontend, k6 facilitates effective load testing by providing an efficient, browser-independent environment that targets HTTP APIs, thus conserving resources. The tool's ability to simulate load with virtual users and validate outcomes through checks and thresholds has proven invaluable in identifying bottlenecks and errors before release, particularly in high-availability setups using Docker Compose and databases like MySQL or Postgres. k6's integration with CI/CD tools and the availability of a Grafana Cloud plugin enable continuous performance monitoring and proactive issue resolution, further evidenced by its role in optimizing database queries and protecting against DDoS attacks. As k6 becomes more integrated with Grafana Cloud, users can seamlessly visualize load test results and metrics, enhancing the overall development workflow and product reliability.

Opsdis, an observability consulting company in Sweden, partnered with Grafana Labs to provide an advanced monitoring solution for Cind AB, a company specializing in 3-D camera technology for measuring objects. Cind initially relied on manual monitoring processes, which required the support team to manually sift through logs when issues arose. To address this, Opsdis implemented a scalable system using Prometheus, Loki, and Grafana, as well as Grafana Cloud, to automate and enhance monitoring capabilities. This new system enables Cind to monitor critical system processes and onsite camera operations, providing comprehensive visibility across over 50 customer sites. The implementation includes templated Grafana dashboards that allow Cind to view customer statuses, analyze metrics, and access logs for troubleshooting. This development not only improved operational efficiency by allowing proactive issue resolution but also integrated observability into Cind's development process, sparking interest in further advancements.

Load testing can be challenging, particularly when scripts produce unexpected HTTP error codes or fail to create the intended outcomes, such as missing database records. Debugging these issues can be complex due to the intricacies of HTTP conversations, especially when dealing with websites rather than APIs and involving dynamic identifiers, necessitating correlation. A web debugging proxy (WDP) can be a valuable tool, acting as a man-in-the-middle to intercept and display HTTP traffic, including encrypted HTTPS data, for analysis. WDPs provide a user-friendly interface to view and compare HTTP requests and responses, which can be particularly useful for identifying problems that aren't evident through console logs or default debugging methods. To use a WDP with the k6 load testing tool, users need to configure specific environment variables to route HTTP/S traffic through the proxy, allowing the examination of both request parameters and responses. While effective, users should disable the proxy for actual load testing due to potential performance issues. The use of WDPs, alongside tools like Postman, can facilitate the creation of accurate and efficient load testing scripts by revealing hidden issues and enabling comparisons with browser-based recordings.

Fiona Liao, a software engineer at Grafana Labs, encountered a bug in Prometheus while working on a Grafana Cloud project involving PromQL query generation. The bug, which led to double-counting of sample values during query evaluation, was discovered through integration tests designed to verify the correctness of generated queries. The issue stemmed from the memSafeIterator in Prometheus, which was not synchronizing correctly with its internal iterator, resulting in erroneous results. Specifically, the Seek() method updated only the internal iterator, causing the memSafeIterator to read incorrect sample values from the buffer. To resolve this, Liao added a Seek() method to the memSafeIterator, ensuring both iterators remained synchronized, and submitted the fix to Prometheus. This experience highlights the importance of integration tests in identifying and resolving software bugs, and Liao encourages interested individuals to consider opportunities at Grafana Labs, where similar challenging and rewarding work is undertaken.

Grafana Labs recently launched its new Grafana Meetup Program with a successful virtual event for the Eastern U.S. & Canada time zone, marking a pivotal moment for community engagement after a year without in-person events due to the COVID-19 pandemic. The event featured several insightful presentations, including Agrology's use of IoT and machine learning to assist farmers in adapting to climate change, with detailed explanations of their sensor network and software application for real-time agricultural insights. Additionally, James Ranson demonstrated the Grafana dashboards designed for Agrology, highlighting features like the "Data Comparator" and future projects to integrate machine learning outputs. Jacob Plicque III provided an introduction to Service-Level Objectives and Indicators, stressing the importance of balancing reliability with other engineering tasks, while Stephanie Closson discussed Grafana's wide time series format introduced in v7.5, which aids in data visualization by grouping data frames by common timestamps. The meetup exemplified Grafana's commitment to fostering community interaction and knowledge sharing, with plans for future events and a potential West Coast Virtual Meetup later in the year.

Grafana 8.1 introduces the Geomap panel, an enhanced version of the Worldmap panel, aimed at offering more dynamic and flexible geospatial data visualization. The Geomap panel allows users to customize world maps with multiple base layer options, including Open Street Maps and ArcGIS, alongside the existing CartoDB tile set. Users can adjust data markers with gradient scales and size variations to represent diverse data points, and six regular shapes are available for markers, with plans to introduce icon options in the future. A heat map layer is available to visualize data density, and the data layer offers mapping options like coordinates, geohash, and lookup to display data flexibly. Additionally, new sharing capabilities enable linking and synchronizing multiple map panels within the same dashboard, facilitating comprehensive data exploration. Overall, the Geomap panel provides a more versatile and interactive experience for visualizing location-based data.

In a typical IoT application with numerous sensors generating data in various protocols, streaming this data to visualization tools like Grafana is crucial for high-resolution monitoring. The introduction of the HTTP Push endpoint in Grafana's v8.0 release under the Grafana Live feature has simplified the process of streaming Telegraf metrics directly to Grafana, enabling real-time data visualization. Telegraf, an open-source server agent known for its plugin-driven architecture, facilitates the collection, processing, and aggregation of metrics from diverse sensor systems and stacks. This demonstration uses an IMU-based sensor system with the MQTT protocol, showcasing how Telegraf can periodically scrape data and seamlessly push it to Grafana Live via the HTTP Telegraf output plugin. This setup is adaptable to various hardware systems utilizing protocols like AMQP, MODBUS, RabbitMQ, and OPC UA, making it a versatile solution for real-time IoT data streaming and visualization.

Nicole van der Hoeven is a Developer Advocate at Grafana Labs, primarily focusing on performance testing at k6. Originally from Melbourne, Australia, she now resides in Maastricht, Netherlands, and has been with Grafana Labs since their acquisition of k6. Her role involves creating load testing scripts and producing content such as blog posts, videos, and presentations. Nicole contributes to open-source projects, including a chaos engineering extension for k6, and enjoys playing Dungeons & Dragons, taking extensive notes, and vlogging about her travels. She values sharing knowledge online to reach a broader audience and finds satisfaction in simplifying technical concepts for beginners. Her desk is dominated by a RØDECaster Pro, an essential tool for her audio and video production. Nicole is multilingual, speaking several languages and choosing travel destinations based on language interests, while she appreciates the authenticity of not using virtual backgrounds during video calls.

With the introduction of Grafana 8.0, streaming real-time sensor data has become more accessible, particularly through the Grafana Live feature, which utilizes a new streaming API. This advancement simplifies the visualization of data from hardware components like sensors and actuators, which are crucial in robotics and automation for monitoring system behavior and diagnosing potential issues. The application of this feature is demonstrated using an IMU-based sensor system that streams data via the MQTT protocol to a Grafana dashboard, showcasing the utility of Grafana Live's publish-subscribe framework. The setup involves a Raspberry Pi and a Bosch Sensortec BMI160 IMU sensor board, with data transmission facilitated by a Python script and a Mosquitto MQTT broker. By leveraging the Grafana MQTT Datasource plugin, real-time accelerometer and gyroscope data are visualized on a Grafana instance, providing dynamic insights into movement and orientation changes. This capability enhances the monitoring of complex systems, revealing imbalances or anomalies through detailed time-series and timeline visualizations, thus underscoring the potential of Grafana's plugin-based architecture for integrating various data streaming protocols and hardware platforms.

Zach Swanson, a Principal Cloud Platform Engineer at Seniorlink, discusses the deployment of Grafana Loki and Grafana Tempo on AWS Fargate without Kubernetes. Seniorlink, which supports families in home caregiving, sought to enhance their service observability by replacing a costly Graylog cluster with Loki and introducing Tempo for distributed tracing. Both Loki and Tempo utilize architecture from Cortex, facilitating scalable data flow and query processing. Despite the Grafana Labs' emphasis on Kubernetes, Swanson's team opted for AWS Fargate for its serverless capabilities, confronting challenges like task configuration and service discovery in the absence of Kubernetes' native tools. By leveraging AWS ECS Discovery and configuring task definitions with Gomplate for templated config files, the team successfully implemented a deployment strategy. They addressed network interface issues specific to Fargate 1.4.0 and emphasized the importance of graceful shutdowns to prevent data loss. The integration of these tools has significantly improved application monitoring and troubleshooting at Seniorlink.

Grafana's Community Plugin Showcase highlights the platform's extensibility through plugins created by both Grafana Labs and its global community, enhancing its utility for monitoring and observability. The Grafana Plugin Directory allows users to easily integrate new data sources, panels, and applications, with recent additions including plugins for Apache Cassandra, gRPC, Apache Druid, Pyroscope, Cumulative Distribution Function (CDF) panels, and Base64 Image/PDF panels. These plugins, developed by various contributors, address specific needs such as real-time analytics, service connection, and visualization enhancements. Grafana encourages developers to create and share their own plugins, offering resources like development documentation and webinars to support them, and invites plugin authors to present their work to the wider community.

Loki 2.3 introduces a new pattern parser for LogQL queries, significantly simplifying the extraction of data from unstructured logs and offering improved performance compared to the regex parser. While previous parsers like JSON and logfmt were efficient, the regex parser was often cumbersome, particularly for tasks like parsing NGINX logs to extract specific labels and values. The pattern parser simplifies this process by allowing users to specify the structure of a log line using a syntax composed of captures and literals, which are more intuitive and faster to execute. Examples demonstrate its applicability for parsing logs from NGINX and Envoy proxy environments, highlighting its ability to efficiently extract relevant fields such as method, status, path, and latency. This new feature, available on Grafana Cloud, aims to make log parsing more accessible and efficient for users, further detailed in the accompanying documentation and GitHub repository.

Loki 2.3 introduces several notable features and improvements, including custom retention settings, enhanced query speed, and Prometheus-style recording rules. The update allows more granular retention control through the Compactor component, enabling storage cost management and compliance with specific security needs. The release also includes experimental support for deleting log entries, with new endpoints for managing deletion requests, and a pattern parser to simplify LogQL queries for unstructured logs. Additionally, ingester sharding boosts query performance up to 15 times for recent data by enabling parallel computation across all time ranges. New LogQL capabilities enhance IP address filtering, and the release is the first under the AGPLv3 license, with easy access options through Grafana Cloud, offering both free and paid plans.

Grafana 8.1 introduces significant enhancements to its open observability platform, building on the features of Grafana 8.0 with new visualization and configuration options. Key additions include the Geomap panel, which replaces the WorldMap panel, and offers custom base tile layers, multiple map layer options, and the ability to define data layers. The new Annotations panel allows for more efficient management of annotations with features like tag-based querying and live filtering. Enhancements to existing panels, such as the Time Series panel, now include annotations and dynamic configuration settings. Grafana 8.1 also brings improvements to logs and traces, including a Prettify JSON option and refined query behavior for trace spans. The release features a new UX for plugin management, time zone support for the MySQL data source, and extended functionality for Elasticsearch. Enterprise users benefit from fine-grained access control, enhanced reporting scheduler flexibility, and the ability to customize Grafana's loading logo for white labeling. Users can explore these features further through the documentation and participate in community forums for discussions.

This blog post outlines an efficient method for managing Prometheus metrics at scale, particularly for users dealing with a large number of individual rule groups. The solution involves leveraging PromQL joins to aggregate metrics more effectively by appending labels from one metric to another at query time, thus eliminating the need for numerous individual rules. The post explains how info metrics can be used to encode high cardinality label values, which can then be joined with other metrics. It highlights the utility of the textfile collector in the Prometheus Node Exporter and Grafana Agent to quickly generate static metrics that are useful for PromQL queries. By repurposing existing tooling to generate a Prometheus metrics file and using a single rule per metric to inject the appropriate reference label, users can achieve the same results as their previous setup but with significantly fewer rules, enhancing both efficiency and manageability.

Grafana Enterprise Logs (GEL) 1.1 introduces label-based access control (LBAC), a feature designed to enhance security by allowing organizations to enforce fine-grained access permissions on log data. This functionality is part of Grafana Labs' broader initiative to provide detailed access control across its products, making it possible for users to limit log access based on specific label requirements, which is particularly useful for protecting sensitive information like personally identifiable information (PII). LBAC, which mirrors a similar feature in Grafana Enterprise Metrics (GEM), enables users to establish access policies that permit log queries only if certain label criteria are met, thereby reducing the amount of data processed and improving query efficiency. This feature underwent five months of development and rigorous testing to ensure reliability, with special attention given to the proper transmission of label matchers across system components. Despite its complexity, LBAC offers flexibility in specifying labels through logical and regular expressions, allowing for precise control over log data access, as demonstrated in Grafana Labs' own billing cluster.

BasisAI utilizes Grafana and Prometheus to monitor model drift in machine learning workloads by employing Boxkite, an open-source Python library that captures and compares data distribution shifts between training and production environments. By integrating with Prometheus, Boxkite generates histograms from both training and production features, which are then exported for Prometheus to scrape and subsequently visualized in Grafana. This setup allows for the detection and alerting of model drift, leveraging metrics such as KL divergence and K-S tests for quantifying drift in categorical and continuous data, respectively. The integration enables a unified approach to monitoring both ML and software performance, allowing teams already using Grafana to seamlessly incorporate ML monitoring through a JSON dashboard, while others can experiment locally with Docker.