May 2021 Summaries

Amnon Heiman's blog post discusses the challenges of monitoring high cardinality data, particularly within systems like Scylla that use Prometheus for metrics collection, and the solutions offered by Grafana Loki 2.0. High cardinality, referring to the large number of distinct metrics, can overwhelm traditional metrics collection systems like Prometheus, which is optimized for handling time series data with lower cardinality. Loki 2.0 introduces features that allow for alert generation based on log data, providing an alternative approach to managing high cardinality by combining logs and metrics. This integration enables the monitoring of specific events with high cardinality by using a low cardinality metric to identify occurrences and then switching to Loki for detailed information. The post emphasizes the complementary strengths of Prometheus in storing metrics and Loki in parsing extensive log data, advocating for their combined use in the Prometheus-Grafana monitoring stack to effectively manage high cardinality scenarios.

Transforming a home office into a Network Operations Center (NOC) room using Philips Hue lights and Grafana is an innovative project that leverages technology to visualize system monitoring. By utilizing Grafana's webhook feature, users can send HTTP notifications of state changes to a custom endpoint, which then communicates with the Philips Hue Bridge. This setup allows the lights to turn red for alerts and green for a normal status, providing a visual representation of system states. The project includes a simple Node.js application available on GitHub to interact with the Hue Bridge APIs, and it uses TestData DB as a Grafana data source for generating test data. A Docker Compose file is provided to easily set up a local Grafana instance with the necessary configurations, and users are guided through setting up authentication with the Hue Bridge. This approach offers a practical example of integrating home automation with network monitoring, enhancing the observability experience with visual cues.

At PromCon, Tom Wilkie from Grafana Labs outlined the future enhancements for Prometheus remote write, a protocol designed to send data from Prometheus to other systems. Key developments include improving metadata transmission, introducing exemplars, achieving atomicity in data transmission, better handling of 429 responses, and reducing bandwidth usage. The goal is to enhance standardization and interoperability among various vendors supporting remote write, ensuring a seamless user experience. Wilkie's team aims to address current challenges such as data loss during outages and the inefficiencies in bandwidth usage due to repetitive label transmission. These initiatives are part of a broader effort to make Prometheus remote write more reliable and efficient, facilitating global federation and integration with other systems.

Cosmote, Greece's largest mobile network provider, leverages Grafana Enterprise to enhance observability in its billing systems, transforming previously time-consuming manual processes into efficient, real-time data visualization. This shift has enabled Cosmote to track business metrics, revenue, expenses, and potential fraud, with automated alerts for any anomalies. Initially adopted by the billing team, Grafana's capabilities have expanded across five business and IT teams, involving at least 35 individuals who monitor various dashboards. Beyond financial monitoring, Grafana played a crucial role during the Covid-19 pandemic by facilitating real-time tracking of government SMS notifications about vaccination campaigns. The integration of Grafana has been pivotal in driving Cosmote's revenue, improving business operations, and increasing company-wide visibility, with a senior analyst likening its impact to "a light" in an otherwise dark cave.

Mikhail Volkov discusses the integration of Redis and Grafana through new plugins that enhance the monitoring and visualization capabilities of the Grafana platform. Redis, a widely used open-source in-memory data store, is employed by many companies for various applications, while Grafana is a popular visualization and analytics platform known for its dashboard capabilities. Previously, connecting Grafana to Redis required using Prometheus as a bridge, which had limitations in terms of metrics and data types. The introduction of new Redis plugins for Grafana simplifies the connection process, allowing users to directly integrate Redis as a data source. This development enables more comprehensive visualization of Redis data, including metrics and modules, in real-time dashboards, without the need for multiple applications or data exports. The plugins support various Redis modules like RedisTimeSeries and RedisGears, facilitating use cases such as financial forecasting, transaction monitoring, and real-time data pipeline visualization. The suite of plugins promises a streamlined workflow and broader data visualization options, underscoring its potential to enhance the observability of Redis data in Grafana.

The updated Strava plugin for Grafana, version 1.3.0, enhances the ability to visualize personal athletic data collected by Strava, a popular service for tracking workouts such as running and cycling. This version introduces improved query options, allowing users to extract and display specific statistics like heart rate, speed, altitude, and more in graphs and tables, tailored to particular activities. New features include the "Fit to Range" toggle for aligning graphs with current time ranges, stats queries for quick data reference, and template variables for constructing reusable dashboards. Additionally, the plugin supports extended stats in table views and includes data links for easy navigation to activity dashboards. The updated plugin also comes with several pre-built dashboards, simplifying the process of getting started, especially on Grafana Cloud, which offers both free and upgraded paid plans.

Grafana Cloud has introduced a new Kubernetes integration aimed at simplifying the monitoring and alerting process for Kubernetes clusters by utilizing the Grafana Agent, which is optimized for collecting and transmitting metric, log, and trace data. This integration streamlines the previously complex setup that required configuring Prometheus Operator or the Grafana Agent, making it easier and less error-prone. Users can deploy a pre-configured Grafana Agent to scrape kubelet and cadvisor endpoints with a few clicks, enabling monitoring and alerting on a range of metrics such as network bandwidth, CPU usage, memory usage, and persistent volume metrics. The integration also includes multi-cluster views, six pre-configured dashboards, and one recording rule to enhance observability. Grafana Cloud offers both free and paid plans, encouraging users to engage with the community and provide feedback on further improvements through the Grafana Labs Community Slack.

The Grafana Enterprise data source plugin for SAP HANA® allows users to integrate and visualize SAP HANA® data within the Grafana platform, breaking data silos and providing a unified view of data sources. This plugin features a built-in query editor, supports complex annotations, and enables alerting, access control, and permissions. Users can monitor production line metrics, such as machine productivity and quality, by setting up template variables and using SQL queries to extract and visualize data, including time series data from sensors. By incorporating annotations, users can correlate events across data sources, aiding in the identification of issues, such as increased machine temperature due to changes in raw materials. The plugin is available for Grafana Cloud and Grafana Enterprise users, and the post encourages feedback and further exploration of the plugin's capabilities through the Grafana Labs Community Slack or by joining the Grafana Labs team.

The first-ever virtual Service Level Objective Conference (SLOConf) emerged from a Twitter conversation among the site reliability engineering community, aiming to enhance customer experience and bridge the gap between engineers and product managers. Grafana Labs is prominently featured at SLOConf, with team members like Richard "RichiH" Hartmann and Björn "Beorn" Rabenstein addressing various aspects of SLOs through on-demand video sessions. Hartmann emphasizes the importance of making non-engineers understand the value of SLOs by aligning them with organizational goals, while Rabenstein critiques both request-based and time-based SLOs for their respective shortcomings. Additionally, Milan Plzik discusses the Production Readiness Review process as a means to bolster confidence in SLOs by identifying and eliminating common pitfalls. Grafana Labs' significant participation in the event reflects its commitment to improving service measurement and infrastructure, and the company is actively seeking individuals who share this passion.

IoT Day at GrafanaCONline, scheduled for June 15, features five sessions focused on the intersection of Grafana and the Internet of Things (IoT), showcasing innovations and practical applications. Kicking off with a demo by Ryan McKinley and Atif Ali, attendees will explore Grafana's enhancements for industrial and IoT use cases, including real-time streaming and operational dashboards. Ed Welch and Ivana Huckova will highlight DIY IoT projects, while Siemens Mobility will discuss using Grafana for near-real-time train sensor data. Grant Pinkos of American Metal Processing will share how Grafana revitalized their manufacturing processes, and Mat Schaffer from Safecast will present on utilizing Grafana dashboards for global radiation and air quality monitoring. The event is open for free registration, offering a comprehensive agenda for IoT enthusiasts.

Grafana Explore facilitates the correlation of metrics and logs by transforming Prometheus queries into Loki queries, and Grafana 8.0 aims to extend this capability to Graphite metrics as well. While Prometheus and Loki have similar query syntaxes, mapping Graphite metrics to Loki requires additional setup due to their differing syntax. This involves configuring how parts of the Graphite metric names should be transformed into Loki label values, as demonstrated through examples of application requests and environment data. Graphite metrics are structured hierarchically, with nodes representing various metric components that can be mapped to Loki labels. The process is simplified with Graphite tags, which allow direct transformation to Loki queries without extra configurations. Additionally, Grafana offers free and paid plans with more features to be revealed at GrafanaCONline, an event providing insights into Grafana 8.0's new capabilities.

Grafana Labs experienced significant growth over the past year, hiring more than 250 employees during the challenges of the global pandemic, which included travel restrictions that meant not meeting over 60% of the team in person. Despite being a remote-first company from the beginning, maintaining the company's culture and priorities amidst rapid expansion and external disruptions has been a key focus. Grafana Labs' efforts have been recognized by Forbes as a 2021 Best Startup Employer and by Inc. as one of the best places to work for the second consecutive year. The company prioritizes building a workplace that employees are proud of over merely reaching headcount targets, and it invites potential candidates to explore opportunities on its updated careers page.

Cortex, developed by Grafana Labs, has evolved to enhance scalability and isolation for Prometheus through innovations such as shuffle sharding. Originally designed to centralize observability and accommodate multiple tenants in a single, scalable cluster, Cortex uses a distributed system to replace the need for a global federation server. Shuffle sharding, inspired by Amazon's techniques, improves tenant isolation by assigning random sub-clusters within the larger cluster, allowing for better fault tolerance and reduced outage risk. This method enables efficient load distribution while maintaining tenant isolation, crucial for managing varying tenant sizes and ensuring robustness against node failures. As Cortex scales to accommodate hundreds of nodes, shuffle sharding has helped minimize outages and isolate tenants effectively, reducing the impact of potential issues like poisoned requests. Additionally, Grafana Labs has enhanced Cortex with features such as query federation and block storage, and as of March 2022, has shifted focus to Grafana Mimir for long-term metric storage.

GrafanaCONline 2021, set for June 7-17, promises to be the largest Grafana community conference ever, coinciding with the release of Grafana 8.0. This virtual event will feature over 30 sessions with 70 presenters, showcasing applications of Grafana in diverse areas such as monitoring the economic impact of the pandemic, aiding space research on the International Space Station, and supporting extensive IT infrastructure. The conference will include workshops on Grafana, Prometheus, and Loki, with increased capacity to accommodate more participants. Key sessions will explore Grafana's role in the observability stack at Dapper Labs, highlighting its integration with Grafana Loki, Grafana Tempo, and Cortex. Enthusiasts are encouraged to register for free and explore the full schedule to engage with the global Grafana community and learn about the latest developments.

Loki is a tool designed for searching logs within systems, particularly useful during incidents or development troubleshooting. While LogQL's line filters in Loki are typically case-sensitive, making it challenging to search logs with unknown capitalization, users can leverage regex line filters such as !~ and |~ with the RE2 syntax for case-insensitive searches. This method allows for efficient searches without performance drawbacks, as Loki optimizes these regex expressions to perform simple case-insensitive searches. To explore Loki's capabilities, users are encouraged to sign up for Grafana Cloud, which offers an accessible platform for managing metrics, logs, and dashboards with both free and paid options.

In the blog post, Daniel González Lopes, a Site Reliability Engineer at k6.io, introduces foobar, a Python-based demo application designed to facilitate the understanding and implementation of distributed tracing using Grafana Tempo. Distributed tracing, crucial for microservice architectures, involves tracking requests across services, which can be complex to set up. The foobar demo consists of two services, foo and bar, both instrumented with OpenTelemetry and capable of exporting traces to a Grafana Tempo backend. The post provides a step-by-step guide on setting up the demo using Docker, verifying its functionality with HTTP requests, and using k6, an open-source load-testing tool, to conduct smoke tests. Unexpected issues arise during testing, revealing the bar service's intentional random failures, which are traced and diagnosed using Grafana Tempo. The post concludes by encouraging readers to explore more about distributed tracing, k6, and Grafana Tempo through available resources and tutorials.

KubeCon + CloudNativeCon Europe 2021 offers a virtual platform for cloud-native enthusiasts to engage with industry experts from Grafana Labs, who will discuss advancements in projects like Cortex, Prometheus, and Jaeger. Throughout the event, team members will present on various topics, including the multi-tenancy and scalability features of Cortex, updates from the CNCF SIG Observability group, and the evolution of Cortex from a vendor-driven to a community-driven project. Attendees will also learn about Jaeger's distributed tracing capabilities and the benefits of the "shuffle sharding" technique for Cortex's scalability and isolation. The event will highlight new features and future improvements for Prometheus, encouraging community involvement and collaboration across the CNCF ecosystem.