February 2022 Summaries

Grafana Labs has developed cardinality management dashboards to aid Grafana Cloud users in monitoring and managing metrics cardinality, a crucial aspect for maintaining efficient system performance and cost management. These dashboards, which include Overview, Metrics, and Labels dashboards, allow users to analyze data from a broad overview to specific metrics or labels, helping identify and potentially reduce high cardinality issues. By offering insights into which metrics and labels are most impactful, users can make informed decisions about retaining, removing, or reducing these elements. For instance, users can limit the number of label values or store data as logs to manage cardinality effectively. The dashboards facilitate understanding of data source origins and imbalances, providing strategies to streamline metrics and labels for optimal use. This tool emphasizes the importance of storing only necessary information, with the flexibility to introduce new labels or values if needed. Grafana encourages users to provide feedback on the dashboards' utility and offers a free trial of Grafana Cloud Pro, promoting its capabilities for enhanced observability.

Grafana Loki version 2.4 introduced a Promtail Kafka Consumer feature that facilitates the ingestion of Kafka messages into Loki for storage, querying, and visualization, enhancing distributed streaming data architectures. This innovation allows logs, traditionally viewed as textual data lines with timestamps and metrics, to be sourced from various platforms like Kubernetes Pods and Windows Event Logs. By leveraging Kafka's scalability and Grafana Loki's flexibility, users can explore new use cases such as IoT data management. The integration includes a setup with four containers: a Kafka Producer and Broker, Promtail, and Zookeeper, enabling efficient message production and consumption with scale and redundancy. While potential message order discrepancies are managed by Grafana Cloud Logs' one-hour time window, users can visualize the data using Grafana dashboards with various LogQL queries. To begin, users can clone a GitHub repository containing all necessary components, and Grafana Cloud offers a free trial for those new to Grafana Loki.

Grafana has introduced exemplar support in Grafana Cloud, enhancing its capability to link metrics with traces, which simplifies identifying performance issues by providing direct trace access from metric data. This integration extends the functionality previously contributed to the Prometheus ecosystem, allowing users to seamlessly pivot from metrics to detailed traces, exemplified by specific data points. Exemplars work by attaching trace IDs to metrics, enabling a straightforward transition from observing high latency in metrics to examining specific traces for root cause analysis. The implementation requires applications to emit traces and include trace IDs in metrics, with Grafana Cloud offering a streamlined setup that eliminates the need to maintain open-source Prometheus, Grafana Loki, and Grafana Tempo. For Grafana Enterprise Stack users, exemplar support will be available in the next release, facilitating similar functionality with minimal configuration changes. This development aims to improve observability by creating stronger connections between metrics, logs, and traces, ultimately aiding in faster incident resolution.

Grafana is a versatile tool that integrates data from numerous sources into a single dashboard, making security a critical concern for its users. To secure Grafana instances effectively, a set of security principles known as the 4 A’s—Authentication, Authorization, Audit, and Administration—plus Encryption, are recommended. Authentication options range from basic to advanced methods like LDAP and SAML, depending on organizational needs, while Authorization uses roles and permissions to control access to dashboards and data sources. Grafana Enterprise offers enhanced features for both authentication and authorization, such as advanced data source permissions and fine-grained access control, which allow more precise management of user actions. Auditing is vital for tracking user activities and ensuring compliance, with Grafana Enterprise providing detailed logs and integration with Grafana Loki for comprehensive audit trails. Administration of these security measures is streamlined through the Grafana interface, supporting the tool's role as a single pane of glass for both data visualization and security management. Encryption further safeguards data by securing communications and protecting sensitive information, with support for various key management systems. This multifaceted approach ensures that Grafana can be secured effectively against unauthorized access and other potential vulnerabilities.

Mihaela Maior, a Senior Engineering Manager at Grafana Labs, plays a key role in leading the Logs & Traces and Observability Experience Squads on the Grafana Experience team, focusing on enhancing the Explore page and observability dashboard experience. Based in Manchester, UK, she values the company's people-centric culture, which emphasizes autonomy, development-driven roadmaps, and self-driven work assignments. Her recent work on restructuring the Observability team reflects her commitment to a people-focused approach, which has garnered positive feedback. Although her role limits coding opportunities, she actively participates in Grafana Hackathons to engage with the codebase and contribute to open-source projects. In her free time, Mihaela enjoys binge-watching TV shows, crafting, and spending time with her family, balancing it with a routine that maximizes her productivity in a remote working environment. She is known for spreading positivity through emojis in her communications, appreciating her team's hard work, and is eager to continue contributing to Grafana Labs’ success.

Grafana 8.4 introduces a range of enhancements aimed at improving usability, performance, and security, with features designed to cater to both novice and expert users. This release includes new panels and charts to facilitate data visualization, enhanced accessibility features for improved navigation and usability, and a full-range log volume histogram for better log data analysis. Performance is bolstered through query caching, which reduces load times and resource use, while security is strengthened with fine-grained access control and improved encryption options. Additionally, Grafana 8.4 offers enhanced alerting capabilities with custom grouping and mute timings for alerts, and supports SAML integration for flexible role assignments across different organizations. These updates are accompanied by thorough documentation and community support to assist users in integrating these new features effectively.

Grafana Labs introduced the Grafana k6 Cloud for Education program, which provided a free k6 Cloud subscription for educators and community members conducting courses and workshops on performance testing using Grafana's open-source tool, k6. Although the program is no longer available, it was designed to help streamline the process of teaching performance testing by simplifying infrastructure management, result interpretation, and group organization. The initiative aimed to alleviate the logistical challenges educators face in preparing courses and to foster hands-on learning experiences. Grafana Labs continues to support educational efforts and plans to release a comprehensive k6 workshop to further empower educators and enhance the learning experience for students and community members interested in performance testing.

Cardinality spikes in observability systems, particularly in Prometheus, refer to sudden increases in the number of unique metric series, which can significantly impact system performance and resource usage. Cardinality, defined as the number of elements in a set, becomes problematic when a metric with low or medium cardinality suddenly shifts to high cardinality, leading to increased data generation and potential system overloads. This can cause memory errors, system crashes, and higher operational costs, as observed with billing in Grafana Cloud Metrics based on the number of active series. Such spikes often occur when new metrics are instrumented with excessive contextual labels like "user_id," creating numerous series for a single metric. Effective cardinality management is crucial for maintaining system efficiency and controlling expenses. Grafana Labs provides resources and tools to help users understand and manage cardinality, offering dashboards and configurations for monitoring and optimizing metric usage.

Grafana Labs is offering a series of free webinars throughout February aimed at enhancing users' understanding and use of Grafana Cloud, Grafana OnCall, Grafana Tempo, and the broader Grafana Stack, which includes Prometheus and Grafana Loki. These webinars are designed to address common challenges in observability, such as managing metrics, logs, and traces, as well as improving on-call management and incident resolution. Participants will learn how to effectively utilize Grafana Cloud for a fully managed observability experience, leverage Grafana Tempo for distributed tracing without complex setups, and use Grafana OnCall for streamlined incident management. The sessions will cover practical demos and best practices, helping users to better visualize and correlate data, thereby strengthening their overall observability strategy.

Antonio Berben and Kim Uyttebrouck's blog post provides a detailed guide on transforming raw data from Istio access logs into valuable metrics using Promtail and visualizing them through Grafana Cloud. It explains how to set up a system architecture for an e-commerce platform selling socks, focusing on how customers filter products by categories like "formal" or "geek." The process involves setting up a Kubernetes cluster, creating a Grafana Cloud account, installing necessary tools like Istio and Prometheus, and configuring Promtail to collect and push metrics to Grafana Cloud. The blog highlights the importance of understanding customer preferences through data visualization, which can help make informed business decisions. Additionally, it suggests using the Grafana Agent as an alternative method for achieving similar outcomes and emphasizes the use of metrics for both analytics and system understanding in DevOps contexts.

Bram Vogelaar, a DevOps Cloud Engineer, discusses the importance of a multi-data center observability stack to ensure high availability and resilience against disasters, such as the fire at an OVHcloud data center that caused significant outages. He describes the process of evolving a single observability stack to a cloud-native, multi-data center setup, using tools like Grafana Tempo and Loki for tracing and logging, while integrating Consul for service discovery across data centers. The configuration includes utilizing MySQL for database replication, employing Consul for Prometheus service discovery, and setting up Grafana Agent to proxy data to multiple Tempo instances. Vogelaar emphasizes the need for configurations that allow seamless failover between data centers, ensuring continuous observability without service interruption, and highlights the collaborative aspects of the Grafana community for improvements and knowledge sharing.

Grafana Labs' Edge team is enhancing real-time hardware observability by integrating new features into Grafana, focusing on applications like autonomous drones and IoT devices. Traditionally, Grafana did not support real-time sensor data ingestion and manual remote control, which are crucial for physical observability. The team developed a simulation-based application using a DJI Mavic 2 Pro drone to test these features, utilizing Grafana Live to visualize real-time telemetry data like accelerometer and GPS information. The project also incorporated real-time drone location mapping and remote control capabilities via a dashboard. This demonstration highlights how Grafana's observability tools can transform swarm-based asset management and tracking, making it easier to leverage sensor data for automation and robotics. With advancements in Grafana Cloud and a variety of plugins, deploying effective observability techniques is becoming increasingly accessible.

Grafana has released versions 8.3.5 and 7.5.15 to address medium-severity security vulnerabilities, including XSS, CSRF, and IDOR vulnerabilities, impacting various versions of the software. The XSS vulnerability, identified as CVE-2022-21702, could allow unauthorized data access through compromised data sources, while the CSRF vulnerability, CVE-2022-21703, enables privilege escalation through cross-origin attacks. The IDOR vulnerability, CVE-2022-21713, affects Grafana Teams APIs, potentially exposing unintended data. Affected users are advised to upgrade their installations promptly, with patches already applied to Grafana Cloud and notifications sent to cloud providers like Amazon Managed Grafana. The vulnerabilities were responsibly disclosed by security researchers, and Grafana encourages further reporting of potential security issues via their designated communication channels.

Grafana Loki, originally designed as a logging backend compatible with Prometheus, offers versatility beyond its intended use by allowing users to store and analyze diverse data formats, as demonstrated in its application for Twitter data analysis. Unlike traditional log indexing systems, Loki indexes only the metadata labels, facilitating acceptance of various log formats and enabling users to stream and query non-log data such as tweets. By utilizing the Twitter API, users can stream tweets into Loki, analyze them using LogQL queries, and track metrics or trends over time. This flexible approach enables exploration of data beyond Twitter, including platforms like Reddit and Salesforce Chatter, highlighting Loki's adaptability for various streaming data sources. With its ability to store plain text and support for customizable, scalable analysis, Grafana Loki proves to be a cost-effective solution for a wide range of data analysis needs.

In September 2021, the U.S. Department of Defense (DoD) authorized the use of Grafana, Grafana Enterprise, and Grafana Loki, streamlining the deployment process for its 100,000 employees and contractors by eliminating the need for additional approvals. In a session at ObservabilityCon 2021, former U.S. Air Force Chief Software Officer Nicolas Chaillan and Grafana Labs’ VP of Global Solution Engineering Steve Mayzak discussed how the DoD utilizes the Grafana stack across various environments, including air-gapped systems and jets, by collecting telemetry data at both granular and global levels. They highlighted the importance of hybrid edge computing, noting that flexible models are essential for adapting technology to diverse military hardware such as submarines and jets. As AI and machine learning evolve, building trust in these technologies is crucial, and the DoD's observability strategy includes developing new training and management approaches. The discussion also touched on the ethical implications of open-source technology and the balance between monolithic and microservices architectures, emphasizing that the journey towards advanced observability is just beginning.

Grafana's time range selector is a crucial feature for visualizing time series and logs, allowing users to choose between absolute, relative, and semi-relative time ranges. Semi-relative time ranges, which combine a fixed start time with a relative "now" end time, are particularly useful for monitoring events that happen over time while retaining historical context. Two practical applications of this feature include troubleshooting infrequent issues, such as random reboots in a network device, and real-time monitoring of metrics updated during specific work hours, like tracking Firefox tab usage throughout a business day. These semi-relative time ranges enable users to efficiently analyze data trends over extended periods without losing sight of the entire history or wasting screen space on irrelevant data. The article encourages exploration of Grafana's flexible visualization capabilities and invites users to share their own experiences and suggestions for using semi-relative time ranges.

Grafana OnCall has been launched for general availability on Grafana Cloud, offering a comprehensive on-call management tool that integrates with Slack, Telegram, voice, and SMS for alerting, and supports existing alerting setups through integrations with services like Datadog and AWS SNS. Initially announced in beta at ObservabilityCON, Grafana OnCall now includes features such as automatic escalations, multi-team support, and a central incident view, all within the familiar Grafana UI. The tool also introduces flexible escalation chains and improved calendar guardrails, and it's available on both free and paid plans, with pricing based on active users rather than provisioned ones. Additionally, a preview of Grafana Incident has been introduced, aimed at simplifying incident response management by automating tasks and providing dedicated workspace for teams. Users can try Grafana OnCall across all Grafana Cloud tiers, with a free trial available for those new to Grafana Cloud.

Grafana Labs has launched Grafana Incident, a tool designed to enhance incident management within Grafana Cloud, available at no additional cost. Grafana Incident aims to streamline coordination and communication during incidents by automatically setting up collaboration spaces, assigning roles, and managing tasks. It maintains a timeline of events and allows for embedding Grafana graphs and other resources to provide a comprehensive view of an incident. Developed from best practices drawn from various industry experiences, including Google and Cloudflare, it features a chatbot interface called Cheesebot that facilitates incident management without a browser, automatically processes URLs, and suggests related dashboards based on context. Grafana Incident is currently in preview, with plans for further integration with Grafana apps and third-party tools, and it is intended to help teams formalize and improve their incident response processes.

The advanced guide to network monitoring with Grafana and Prometheus delves into the use of SNMP (Simple Network Management Protocol) and the snmp_exporter for effective monitoring of network infrastructure devices such as switches, routers, and firewalls. SNMP is a standard protocol that, despite its name, is complex and involves collecting data from network devices using Management Information Bases (MIBs). The guide explains how the snmp_exporter facilitates the exposure of SNMP-collected information for use by the Prometheus monitoring system, outlining the configuration process involving a Prometheus server, the generator for parsing MIBs, and the snmp_exporter itself. Emphasizing the importance of surgical precision in selecting Object Identifiers (OIDs), the instructions guide users through retrieving MIBs, running snmpwalk for data extraction, generating configuration files, and setting up the snmp_exporter. The document also highlights the integration of these components into Prometheus for data scraping and visualization in Grafana dashboards, providing insights into the nuanced setup process while promoting the use of Grafana Cloud for an accessible start with Prometheus and Grafana.