April 2021 Summaries

PromCon Online 2021, a virtual event held on May 3 as part of KubeCon + CloudNativeCon, focused on the Prometheus monitoring system and featured contributions from the Grafana Labs team among thousands of Prometheus users. Highlights included Grafana Labs' Director of Community Richard Hartmann discussing efforts to make the Prometheus community more open, Peter Å tibraný explaining the conversion of Cortex data to TSDB blocks, and Tom Wilkie exploring the future of Prometheus remote write. Additionally, Björn Rabenstein presented improvements in Prometheus histograms, and Atibhi Agrawal offered insights into new matchers in Prometheus Alertmanager. The event marked Grafana Labs' transition from contributing to Cortex to launching Grafana Mimir, a scalable long-term storage solution for Prometheus, capable of managing over a billion metrics.

Grafana Enterprise Metrics (GEM) 1.3 enhances the Prometheus-as-a-Service experience by improving its plugin to allow more direct management of settings, including hash-ring health monitoring and custom tenant limits. This update facilitates the identification of issues within microservices by integrating hash-ring health status directly into the plugin, making it easier for users to manage and troubleshoot their metrics clusters without relying on disparate API endpoints. Additionally, GEM now supports setting custom tenant limits through the plugin, streamlining the process for centralized observability teams to allocate resources efficiently between different application teams without the need for cumbersome configuration file updates. These updates aim to provide a more efficient and user-friendly experience for managing multi-tenant environments, with plans for further enhancements in visualizations and cluster health indicators.

The Elixir PromEx library, developed by Alex Koutmos, is designed to seamlessly integrate Prometheus metrics with Grafana dashboards, specifically catering to the Elixir and Erlang ecosystem. PromEx utilizes the BEAM Telemetry tools to consistently capture and surface internal library events, allowing users to attach Prometheus metrics collectors to these events through a plugin-based framework. With PromEx, users can quickly set up dashboards for various Elixir libraries like Phoenix, Ecto, and Oban, leveraging the Grafana HTTP API to streamline dashboard deployment and lifecycle management. This integration not only automates the visualization of metrics but also addresses common challenges such as manual dashboard updates and environment-specific configurations, ensuring that the lifecycle of dashboards aligns with application versions. PromEx aims to expand its capabilities by adding more plugins and integrating with the OpenTelemetry Erlang/Elixir project, highlighting its potential applicability beyond the Elixir ecosystem.

Grafana Labs conducted performance tests on Grafana Enterprise Metrics (GEM), a self-hosted Prometheus service, to evaluate its horizontal scalability and performance under high loads. By developing the benchtool, they tested GEM with up to 500 million active series, demonstrating its ability to handle this scale with linear hardware usage. The tests showed that GEM maintained a write path latency of less than 3.1 seconds at the 99th percentile and successfully handled over 99% of query requests, with median query latency remaining under 1.35 seconds. Additionally, the alternative KV store, memberlist, proved capable of managing over 500 ring members, offering flexibility in environments without Consul. Future testing will focus on the GEM compactor and store-gateway throughput to further enhance performance for queries involving older data. These findings underscore GEM's robustness and scalability in handling large-scale metrics workloads.

PayIt, a secure cloud service provider for digital government, has been utilizing Grafana and Prometheus since 2015 to enhance its observability and alerting capabilities as it transitioned from a single Kubernetes cluster to a more complex microservices-based architecture. Grafana's dashboards have enabled PayIt to efficiently monitor its systems, helping to solve user-facing issues and providing data accessibility to non-engineers, thus streamlining support operations. By leveraging community dashboards and creating custom ones, PayIt has improved its ability to visualize data and identify potential problems, allowing the company to prevent customer-facing issues and optimize internal processes. The adoption of Grafana has opened up new possibilities for PayIt, with the team exploring further uses of the tool to maximize its potential benefits.

Dan Jensen, the Senior Manager of Public Relations at Grafana Labs, is based in Seattle, Washington, and is responsible for communicating the company's stories to media and external audiences. He joined Grafana Labs due to its strong community and passionate team. In his leisure time, Jensen enjoys skiing, golfing, attending sporting events with his family, and using his Sonos speaker setup for music throughout the house. He finds music essential for productivity, tailoring playlists to match his daily mood. Recently, he binge-watched "Ted Lasso" and has taken up smoking meats as a new hobby. Looking forward to post-Covid activities, he anticipates enjoying a dinner and movie outing with his wife. For working from home, he advises setting personal time on the calendar and avoiding lunch at the desk to maintain a healthy work-life balance.

Grafana Agent 0.14.0-rc2 introduces enhanced support for Windows, addressing previous challenges such as lack of native Windows Service integration and difficulties with the windows_exporter. The update enables Grafana Agent to run as a native Windows Service, eliminating the need for external shim software and allowing seamless response to start and stop commands. It also integrates windows_exporter to streamline dependency management and improve metrics collection. Additionally, the release includes a new installer and uninstaller, offering default configuration setup, silent installation, and protection of user configurations during updates. Grafana Agent logs can now be viewed through the Windows Event Log, enhancing observability and integration with existing workflows. The Grafana team invites feedback via GitHub to further refine the Windows experience.

A live introductory webinar on Prometheus, led by project maintainer Goutham Veeramachaneni, is scheduled for April 22, offering an overview of the open-source project widely used for monitoring Kubernetes and cloud-native systems. The session will demonstrate how to easily start using Prometheus with Grafana Cloud, a composable observability platform, highlighting the efforts made by the Grafana Labs team to simplify Prometheus adoption for beginners. Attendees will learn how to quickly implement metrics and alerts, and can register for the event online. For those not yet using Grafana Cloud, new free and paid plans are available to accommodate various needs.

Grafana Labs CEO Raj Dutt discusses the company's decision to relicense its core open-source projects, including Grafana, Grafana Loki, and Grafana Tempo, from Apache 2.0 to AGPLv3. This move aligns with their commitment to maintaining transparency and adhering to OSI-approved licenses while emphasizing the importance of community support. Although AGPLv3 is expected to create potential complexities for some users, Grafana Labs believes that most companies will adapt without significant issues. The change will not impact their strategic partnerships, such as with AWS, and they aim to provide guidance for users on compliance with the new license. Dutt also highlights the importance of maintaining a strong community and brand presence to mitigate risks associated with potential forks of their projects. The conversation underscores Grafana Labs' dedication to being an open-source leader while balancing commercial success and community health.

Grafana Labs, founded in 2014 to support the open-source Grafana project, has decided to relicense its core projects—Grafana, Grafana Loki, and Grafana Tempo—from the Apache License 2.0 to the Affero General Public License (AGPL) v3. This decision follows a trend among other open-source companies to adopt more restrictive licenses to balance open-source community values with monetization strategies. The AGPLv3, an OSI-approved license, maintains the freedoms of free and open-source software while requiring users to share source code if they modify and distribute it, encouraging contributions back to the community. Although AGPL offers less protection than other licenses, Grafana Labs believes it strikes the right balance in fostering a stronger community. Additionally, a new contributor license agreement based on the Apache Software Foundation's CLA has been adopted to clearly outline the terms between contributors and Grafana Labs.

Amazon Web Services (AWS) and Grafana Labs have expanded their collaboration by releasing a new Open Distro for Elasticsearch plugin for Grafana, available as part of the Amazon Managed Service for Grafana. This managed service allows users to integrate various data sources such as ServiceNow, Splunk, and Snowflake into Grafana while providing the option to upgrade to the full Grafana Enterprise experience, which includes expert support and training. The new plugin is designed for users of Amazon Elasticsearch Service and Open Distro for Elasticsearch, enhancing Grafana's plugin ecosystem with features such as enterprise security and the Piped Processing Language (PPL) for complex queries. This plugin complements the previously released Elasticsearch data source plugin developed in partnership with Elastic.co and supports users in efficiently querying and analyzing data within Grafana. AWS and Grafana Labs are hosting a webinar to further discuss these offerings and their benefits.

Austin Freels shares his experience of using a Raspberry Pi, Grafana, and Prometheus to monitor home energy consumption, aiming to identify which household devices consume the most power. He utilized an Emporia Vue 2 energy monitor to gather energy metrics, which are then processed by HomeAssistant and Prometheus before being visualized on a Grafana Cloud dashboard. Freels outlines the step-by-step process, including configuring the Raspberry Pi with necessary software such as Prometheus and Docker, setting up Grafana Cloud for data visualization, and using HomeAssistant to manage energy data. He emphasizes the ease of use and potential cost savings, noting his discovery that his water heater accounts for a significant portion of his energy bill. Freels encourages others to try this setup and offers insights into the benefits of Grafana Cloud's free and paid plans for comprehensive monitoring solutions.

The new Tencent Cloud Monitor App plugin for Grafana enables users to efficiently oversee their Tencent Cloud services, including Cloud Virtual Machines and Cloud Databases, by integrating Tencent's infrastructure capabilities with Grafana's data visualization tools. Developed by the Tencent Cloud team, the plugin capitalizes on the Tencent Cloud Monitoring API (3.0) to pull performance metrics and offers dashboard templates for seamless monitoring experiences. This plugin is compatible with both the open-source and enterprise versions of Grafana, which support various data sources and commercial tools. It simplifies the setup process, requiring only the installation of Grafana, addition of the plugin, and configuration with user credentials to start monitoring. Through this plugin, Tencent Cloud users can enhance their cloud service management by accessing comprehensive data insights on a unified dashboard.

Grafana Cloud's Tempo service, currently in beta, allows users to send traces using open standards and interoperability, offering a flexible solution compatible with various tools. The service is accessible via a gRPC endpoint conforming to the OpenTelemetry TraceService with HTTP Basic authorization, enabling any tool that can handle HTTP headers to send traces. This tutorial demonstrates how to configure the OpenTelemetry Collector for sending traces to Grafana Cloud. Users are guided through gathering necessary credentials, creating and configuring the otel-config.yaml file, and using docker-compose to run the collector and the Jaeger HotROD demo for generating and sending trace data. The post emphasizes the flexibility of using Grafana Cloud through its adherence to industry standards and provides a starting point for those interested in exploring its capabilities, including both free and paid plan options.

The post delves into troubleshooting remote write issues in Prometheus, emphasizing the complexities of its tunable settings and the potential for data loss. Initially, remote write duplicated scraped samples, but challenges with fixed-size buffers led to data drops or memory overloads during disruptions. To mitigate data loss, Prometheus now reads data from its write-ahead log, offering a 2- to 3-hour disk buffer, reducing reliance on large in-memory buffers. The text explains key metrics for diagnosing remote write issues, such as those indicating how far remote write is falling behind or how many shards are active. It also outlines configuration parameters like shard numbers and batch sizes to manage throughput and network load. The post concludes by highlighting ongoing efforts to enhance remote write's reliability and encourages community engagement for feedback and contributions.

Daniel Palay, the Senior Manager of Customer & Community Marketing at Grafana Labs, is dedicated to capturing and sharing success stories from customers and community members. Based in San Jose, California, he emphasizes the collaborative spirit of the Grafana community, which originated to solve issues not addressed by Kibana. Outside of work, Daniel enjoys activities like playing ice hockey, engaging in home improvement projects, and spending time with his wife, Jenny, and their cat, Little Man. He expresses a clear passion for storytelling, particularly in highlighting the joy and achievements of Grafana users at events like GrafanaCON and ObsCON. For personal tech, he enjoys using his iPhone 12 Pro, especially its Portrait mode for photography. Daniel also shares a work-from-home tip, advocating for clear boundaries between work and home life to maintain mental health.

Grafana Labs employs a metamonitoring system using Prometheus servers to monitor other Prometheus servers, ensuring that monitoring failures are quickly identified and addressed. This system involves a network of geographically distributed metamonitoring Prometheus servers that monitor each other across clusters, with a security mechanism similar to a dead-man’s-switch. The setup includes high-availability (HA) Prometheus pairs within Kubernetes clusters, a global Alertmanager cluster, and the use of Vault for managing authentication and secrets across clusters. Alerts are routed through Prometheus to Alertmanager and finally to PagerDuty, with a heartbeat system in place using Dead Man’s Snitch to ensure notification even if the alerting chain fails. This approach ensures redundancy and reliability, allowing Grafana Labs to maintain observability and sound alerts during outages of any part of the monitoring infrastructure.

Raspberry Pi enthusiasts can enhance their projects by using Grafana Cloud for monitoring purposes, as highlighted in a blog post detailing its simplicity and efficiency. The post promotes a live webinar featuring guidance on connecting and monitoring a Raspberry Pi with Grafana, illustrating how easy it is to set up dashboards, metrics, logs, and alerts without the need for a full installation process. Grafana Cloud offers a free tier with 10K series for Prometheus or Graphite metrics and 50GB of logs, along with a library of prebuilt integrations such as the Linux integration, making it an accessible tool for users to start observing their Raspberry Pi projects efficiently.

Kubernetes Community Days Africa, set for April 29, is a pioneering online conference aimed at fostering interest and participation in open source cloud native technologies within Africa. Organized in part by Uchechukwu Obasi, a CNCF Ambassador and frontend software engineer at Grafana Labs, the event seeks to build a vibrant community of cloud native professionals on the continent. The conference will feature two tracks—community and enterprise—catering to both students and newcomers as well as those interested in large-scale cloud native operations. Attendees can expect interactive sessions and workshops on topics like Kubernetes and Grafana, with notable speakers including Google Principal Engineer Kelsey Hightower and CNCF General Manager Priyanka Sharma. Grafana Labs is sponsoring internet access for the event, and interested participants can register for free to join this significant gathering of cloud native enthusiasts.

Visualizing industrial IoT data becomes more accessible and cost-effective by using Telegraf plugins with the Grafana Cloud Hosted Prometheus service, especially in complex systems like Modbus-based PLCs. Traditionally, setting up such visualizations required substantial middleware and additional tools, but the integration of open-source tools such as Prometheus and Grafana simplifies the process. By leveraging Grafana Cloud, which combines metrics, logs, and traces into a single platform, users can utilize powerful observability software without the overhead of managing infrastructure. A demonstration involving a PLC simulator based on Codesys Control SoftPLC shows how data streams, like those from a simulated Refrigerator Controller, can be visualized using Grafana Cloud. The setup uses the Modbus Telegraf input plugin to collect data and the HTTP Telegraf output plugin to transmit it to the cloud. This approach offers a streamlined method for monitoring important parameters and provides a free tier for experimentation, allowing users to explore these capabilities without significant investment.

Natalia Bernarte is an engineering manager for the Grafana frontend platform team at Grafana Labs, based in Berlin, Germany. She supports her team in developing the Grafana open-source project, focusing on creating a user-friendly platform and fostering a collaborative work environment. Although she rarely codes, Natalia contributes to community discussions and participates in team activities like gardening days. She enjoys cooking, especially making Neapolitan-style pizza, and is passionate about Spanish cuisine, with a particular fondness for Spanish tortilla. In her leisure time, Natalia indulges in reading, learning to draw, and binge-watching shows like "Buffy the Vampire Slayer." Additionally, she possesses unique talents, such as memorizing "Harry Potter and the Philosopher’s Stone" dialogues and performing the rare feat of touching her nose with her tongue.

Prometheus, a widely used monitoring system, allows users to define alerting rules using PromQL, helping them determine when and how they should be notified about system metrics. Key strategies include focusing on USE (utilization, saturation, errors) and RED (rates, errors, duration/latency) metrics, with specific alerts such as monitoring service availability, memory usage, disk capacity, and response times. Grafana Labs Solutions Engineering team offers guidance on implementing these alerts, which can help efficiently manage resources and avoid unnecessary alerts. The article highlights the importance of understanding key performance indicators and suggests leveraging tools like Grafana Cloud's Prometheus-style UI and integrations for managing alerts and recording rules.