June 2022 Summaries

Organizations worldwide are at a pivotal moment, deciding between returning to traditional office settings or adopting a remote-first approach, a topic explored in Grafana's podcast featuring Darren Murph of GitLab. Murph advocates for remote-first work environments, emphasizing the focus on outcomes over physical presence, which benefits both individuals and organizations by fostering innovation, resilience, and talent acquisition. He criticizes hybrid models for not addressing core issues of work dynamics and urges companies to invest in remote-first tools and policies. The discussion highlights the importance of strategic in-person interactions to enhance team rapport and culture, suggesting that organizations allocate resources for meaningful face-to-face meetings. Murph encourages individuals to challenge traditional work-life paradigms, viewing the current shift as an opportunity to design a more fulfilling and impactful professional and personal life.

Grafana Labs has enhanced the user experience for its reporting functionality in Grafana Enterprise and Grafana Cloud Pro and Advanced through a cross-functional team approach involving UX, product, and engineering disciplines. The updates, introduced with Grafana 9.0, address previous shortcomings in reporting by making it more powerful and user-friendly, incorporating features like flexible scheduling, multiple dashboard reporting, and new export formats. To improve usability, the reporting process was reimagined as a multi-step wizard, reducing cognitive load and errors while aligning with user workflows. The changes were validated through usability testing, which revealed positive user reactions, and further improvements are planned, such as the ability to save report drafts. Grafana Labs encourages user feedback and provides a free trial of Grafana Cloud Pro for those interested in exploring the updated features.

Grafana 9 introduces Role-based Access Control (RBAC) as a general availability feature, enhancing the platform's authorization system by providing more flexible and secure access management. Originally introduced as a beta in Grafana 8.0, RBAC is now enabled by default and offers seamless transitions for both open-source and enterprise users, allowing assignment of specific roles like Dashboard Writer or Data Source Reader to users, teams, and service accounts. Users of Grafana Enterprise and Grafana Cloud Advanced can further customize access levels and use provisioning tools like Terraform. The update extends RBAC to cover all main Grafana resources, includes audit logs for tracking permission changes, and provides a UI role picker for easy management. However, the 9.0 release also brings breaking changes, such as renaming built-in roles to basic roles and removing permission inheritance, necessitating updates to provisioning files. Future enhancements aim to expand RBAC functionalities, including better scoping and role sync with external authentication providers.

Willie Engelbrecht describes using Grafana Cloud and an ESP32 microcontroller to monitor a home water softener's salt levels to prevent the issues caused by hard water, such as dry skin and pipe damage. After moving to an area with hard water, Engelbrecht decided to create a cost-effective solution instead of purchasing expensive water softeners with built-in sensors. By utilizing Grafana Cloud's capabilities, Engelbrecht set up a system to track salt levels in the water softener using a JSN-SR04T distance sensor and ESP32 microcontroller, which sends data every two hours to Grafana Cloud for visualization. This setup allows Engelbrecht to receive alerts when the salt levels fall below a certain threshold, ensuring timely replacement and preventing the return of hard water. Engelbrecht highlights the integration with Grafana Cloud as an easy way to manage metrics and expresses satisfaction with the project, hinting at future IoT projects.

Running distributed load tests on Kubernetes using the Grafana k6 tool involves deploying and managing k6 test scripts within a Kubernetes cluster, leveraging the Kubernetes operator pattern. This allows for automated management of distributed test jobs, which is beneficial for users who prefer to run tests within their private network or maintain consistency across their Kubernetes-managed infrastructure. The process includes setting up a Kubernetes cluster, deploying the k6 operator, creating and configuring test scripts, and using custom resources to initiate tests. Despite simplifying distributed load testing, users must address challenges like metric aggregation, threshold evaluation, and resource allocation. The k6 operator supports cloud output for metric aggregation, though it increases the overall cost of ownership due to the need for self-hosting and data management. The Grafana Cloud k6 service offers an alternative for those who prefer a managed solution, providing a streamlined approach to performance testing.

The guide details the process of setting up an OpenTelemetry Collector to gather and send logs to Grafana Loki using Grafana Cloud, with specific examples for using Fluent Forward and Filelog receivers. It outlines the prerequisites, including having Grafana Loki and Grafana running, and provides instructions for setting up a Grafana Cloud account, generating an API key, and configuring the OpenTelemetry Collector with the necessary YAML configuration files. The Fluent Forward receiver collects logs from Docker containers, while the Filelog receiver tails and parses system logs, both of which can be viewed in Grafana Cloud's Explore section. The guide also mentions that the OpenTelemetry Collector can simultaneously use multiple receivers and suggests its utility for distributed tracing and metrics collection with Grafana Tempo, encouraging users to try Grafana Cloud for enhanced observability features.

Grafana 9 has introduced a command palette designed to enhance the user experience, especially for power users who prefer using keyboard shortcuts for efficiency. This feature, inspired by similar functionalities in code editors and websites, allows users to perform various tasks such as navigating, searching, and managing dashboards with simple keystrokes, specifically cmd+k for macOS and ctrl+k for Linux/Windows. The command palette is part of a broader range of updates in Grafana 9, including improvements in navigation, search, new query builders for Prometheus and Grafana Loki, and security enhancements. The release aims to make Grafana more accessible and efficient, encouraging users to explore its capabilities through documentation and online resources like GrafanaCONline 2022. Grafana Cloud also offers a free tier to help users get started with metrics, logs, traces, and dashboards.

GrafanaCONline 2022 concluded with notable sessions highlighting advancements in Grafana Labs technical documentation, innovative uses of Grafana Cloud in activist engineering for air quality monitoring, and the extensive capabilities of the Grafana k6 load testing tool. The event also showcased success stories like KCB Bank Uganda's transition to Grafana for improved transaction monitoring, resulting in significant service uptime improvements and revenue growth. Noteworthy presentations included DesignSpark's citizen science project using Grafana Cloud, Cisco's utilization of Grafana for network data visibility at its annual event, and Optum's use of Grafana Enterprise for healthcare website performance monitoring. Additionally, the event featured a hackathon showcase with creative projects such as real-time drone tracking and geospatial data plugins, and Medallia's journey to centralized observability through Grafana dashboards. Participants who missed the live sessions can still access recordings on demand.

Grafana Labs conducts company-wide hackathons three times a year, providing employees a week to step away from their usual tasks to innovate and create freely. These events, averaging 130 participants and producing over 150 projects within eight months, foster a competitive yet collaborative environment where creativity thrives without the constraints of regular work mandates. Hosted by Grafana's podcast, "Grafana's Big Tent," the hackathons encourage cross-disciplinary collaboration and are celebrated with presentations at company all-hands meetings. Projects range from fun experiments, like running Doom on Grafana, to significant contributions added to the product roadmap. The hackathons are gamified with judging and prize money, adding an element of competition, though the main motivator remains the opportunity for creative expression and collaboration. This initiative allows Grafana employees to pursue personal interests, solve practical problems, and maintain the innovative spirit that characterizes the company.

GrafanaCONline 2022 Day 3 provided insights into the latest developments in Grafana 9 and Grafana Loki, highlighting significant enhancements in alerting functions and log management capabilities. The event showcased how Grafana's unified alerting system improves data health scorecards for companies like Torqata, enhancing efficiency in data-driven industries. It delved into Grafana Loki's advancements in scalability and latency reduction, demonstrating its deployment in monitoring battery-operated NB-IoT sensors by Fuelics PC. Presenters shared best practices for creating effective dashboards, discussed Formula 1 telemetry analysis using Grafana Cloud and Azure Data Explorer, and explored Medallia's journey to centralized observability through Grafana dashboards. The event also highlighted various use cases for Grafana's tools, promoting the platform's adaptability across different sectors, from automotive analytics to real-time data visualization in competitive sports.

GrafanaCONline 2022 continued its momentum on Day 2 with engaging sessions highlighting the latest innovations in Grafana 9, Grafana Mimir, and Grafana Tempo. These sessions showcased various new features aimed at improving ease of use, visualization, reporting, and security. Notable demonstrations included machine learning for real-time microscopy image analysis, the transformation of a mini arcade into a Grafana display, and the scalability of Grafana Mimir for handling extensive data series. Additionally, Grafana's flexibility was displayed through projects such as using it as a personal health tracker and for real-time drone tracking. A hackathon presented creative uses of Grafana, including running the game Doom and enriching geospatial data visualization. The event underscored the versatility and expansive utility of Grafana across diverse applications in technology and personal projects.

GrafanaCONline 2022 kicked off with major announcements including the release of Grafana 9.0 and the introduction of Grafana OnCall to the open-source community. The event featured diverse applications of Grafana, such as monitoring satellite TV operations at Dish Network, enhancing astrophotography with Grafana Cloud, and optimizing food waste reduction processes at Apeel Sciences. Other sessions showcased Grafana's use in live stock and crypto trading data visualization, modernizing industrial IoT at Varland Plating, and providing network visibility at Cisco Live. Attendees were encouraged to join live sessions and access on-demand content while engaging with the Grafana community through various platforms.

GrafanaCONline 2022 marked the launch of Grafana 9.0, which focuses on enhancing the user experience for observability and data visualization, introducing new features like Prometheus and Grafana Loki visual query builders, and panel and dashboard search functionality. The event highlighted the integration of Grafana Alerting as the default in Grafana 9.0, offering an improved alerting experience. Grafana OnCall was also open-sourced, enabling self-managed installations for organizations with specific security and connectivity needs. Additionally, Grafana Labs announced the Grafana Live conference series, aiming to connect with users in various cities and showcase the capabilities of Grafana's LGTM open-source stack. The conference offered over 30 sessions with over 50 speakers, featuring live demos and best practices from industry experts.

Grafana 9 introduces a revamped and unified alerting system designed to enhance user experience by integrating feedback from the community, offering significant improvements over the previous versions. This system combines Grafana panel and Prometheus-style alerts into a single, cohesive interface, allowing for the creation of multi-dimensional alerts that can generate multiple instances per rule for greater flexibility and control. Users can now manage alerts more effectively with features such as grouping, routing, and silencing notifications, ensuring that alerts are organized and sent based on specific criteria. The introduction of notification policies allows for sophisticated routing and bundling of alerts to prevent notification overloads, while mute timings enable users to define periods when alerts are not to be sent. The system remains open-source and is available across all Grafana deployments, with plans for further enhancements such as improved user experiences, role-based access control, and an updated Terraform provider. As Grafana Alerting becomes the default in Grafana 9, the option to revert to the legacy system will be phased out in future versions, highlighting the company's commitment to continuous improvement and user-centric development.

Grafana Labs has announced the open-sourcing of Grafana OnCall, an on-call management tool initially launched as a cloud-only solution for Grafana Cloud users. Grafana OnCall, originating from the acquisition of Amixr Inc., was designed to simplify workflows and enhance incident response for DevOps and site reliability engineering (SRE) teams. The open-source version, Grafana OnCall OSS, allows users to self-manage deployments, addressing concerns like security and limited connectivity. By integrating into Grafana’s observability stack, the tool facilitates efficient collaboration and faster incident resolutions, remaining connected through Grafana Cloud’s 24/7 monitoring. The open-source initiative invites community participation in development, offering flexibility in local environments while maintaining access to Grafana Cloud’s features. This move aligns with Grafana’s broader strategy of making its tools accessible and resilient, empowering engineers with reliable notification systems.

GrafanaCONline 2022 is set to take place from June 13 to 17, featuring over 30 sessions with more than 50 speakers from companies like Optum, Dish Network, and Cisco, who will discuss their use of Grafana and other open-source tools. The event will highlight the launch of Grafana 9.0, showcase projects from Grafana Labs' hackathons, and provide demos of key developments in tools such as Grafana Tempo, Loki, and k6. Attendees are encouraged to register for free workshops, view the full agenda, and participate in live Q&As via the event page. Additionally, participants can connect through the Grafana Labs Community Slack workspace and have the chance to win Grafana Labs swag packs. The event is governed by a Code of Conduct to ensure a positive experience for all attendees.

Ivana Huckova shares her innovative project of monitoring a sourdough starter using Prometheus and Grafana Cloud, inspired by Christine Sunu's sourd.io. As a software developer at Grafana Labs, she utilizes an ESP32 development board connected to sensors to track the starter's temperature, humidity, and height, and visualizes the data in Grafana. The setup involves using an ultrasonic sensor for measuring the starter's rise, a DHT11 sensor for environmental conditions, and programming the ESP32 to send data to a Prometheus-compatible database in Grafana Cloud. Huckova details the process of setting up the necessary software and hardware, including installing Arduino IDE and relevant libraries, and provides insights into visualizing the data on Grafana dashboards. She emphasizes the utility of alerts for timely feeding and invites feedback and ideas from the IoT community, highlighting Grafana Cloud's capabilities for metrics and dashboarding.

Grafana k6, a modern load-testing tool, can be used to evaluate the scalability, reliability, and performance of a Loki installation by simulating real-world log ingestion and querying scenarios. This approach allows users to determine the capacity and speed of their log clusters efficiently, without relying solely on existing infrastructure. Grafana k6's scripting API, compatible with local and cloud environments, includes an extension that serves as a Loki client, enabling users to push logs and perform queries while utilizing the combined efficiency of Go code and the flexibility of JavaScript. The tool supports both single-user and multi-tenant modes, offering customizable configurations for label cardinality to generate unique log streams. It also provides a range of metrics for performance assessment, helping users analyze and compare test outcomes. By using these features, users can effectively set up and conduct load tests on their Loki installations, gaining deeper insights into their system's data processing capabilities.

Vernon Miller, a senior software engineer at Grafana, shares his journey of setting up a homelab cluster to enhance his understanding of Kubernetes and Grafana Mimir. Transitioning from private clouds and on-prem infrastructure to public cloud services, he found Kubernetes particularly challenging. To bridge the gap between local development and cloud clusters, he created a physical multi-node cluster, named "beehive," using Beelink Mini PCs instead of the originally planned Raspberry Pi. This setup allowed him to experiment and learn by repeatedly breaking and fixing things. Through this process, he installed a k3s cluster, configured Prometheus and Grafana Mimir using Helm charts, and utilized his TrueNAS server for storage. Although faced with challenges like memory limitations and configuration complexities, Miller gained valuable insights into the operational aspects and scalability of Grafana Mimir, ultimately finding the experience both educational and enjoyable.

CVE-2022-32276 and CVE-2022-32275 were reported as potential vulnerabilities in Grafana, focusing on UI issues that Grafana Labs assessed as non-security impacting. Despite CVE-2022-32275 being assigned a severity score of 7.5 HIGH, Grafana Labs maintains that neither CVE poses a security threat, as both involve the display of the 'not found' page within the regular UI without granting privileged access. Grafana Labs is engaging with MITRE to reconsider the assigned severity score and encourages public discussion on the two open GitHub issues tracking these concerns. The company appreciates the report from the community and invites further security reports through their designated email, emphasizing the importance of confidentiality until a fix is deployed.

Grafana dashboards, while easy to create, can become challenging to organize, especially as organizations grow and users increase their dashboard outputs, leading to potential sprawl. The article provides a comprehensive overview of various types of Grafana dashboards, such as USE/REDS, overview and drill-down, business journey, and R&D dashboards, each serving unique functions from monitoring infrastructure to visualizing business processes. It emphasizes the importance of categorizing and efficiently organizing dashboards using features like folders, tags, and naming conventions, although Grafana's current folder structure is limited to first-level folders only. The discussion also touches on advanced topics like dashboards as code, which allows dashboards to be generated and managed through automated scripts, though this is still an evolving area with potential changes in the future. Grafana provides tools like its HTTP API and provisioning to automate dashboard management, and the article suggests strategies for organizing dashboards effectively, such as creating personal sandboxes or using tags for dynamic dashboard lists. The author shares insights from their own experience and other expert talks, offering guidance on establishing a flexible yet ordered system that accommodates a wide range of dashboard types and use cases.

At KubeCon + CloudNativeCon EU 2022 in Valencia, Spain, Grafana Labs shared insights on advancements in Kubernetes monitoring, Prometheus histograms, and observability. Highlights included the introduction of the kubectl alpha events command to enhance Kubernetes monitoring, insights into the Prometheus ecosystem, and the future of alerting with Prometheus Alertmanager. Sparse high-resolution histograms were demonstrated for their ability to transform data representation and querying in Prometheus. The event also covered the role of CNCF Technical Advisory Groups in improving observability and the challenges of deploying Kubernetes across multiple cloud providers. Grafana Labs showcased its successful multi-cloud deployments and invited participation in future projects and events like GrafanaCONline 2022.

In a podcast episode from Grafana's "Big Tent," hosts Mat Ryer and Matt Toback discuss observability strategies with Nayana Shetty, a principal engineer at the Lego Group, emphasizing the importance of incorporating observability from the outset rather than as an afterthought. The conversation highlights Shetty's experiences with building reliable microservices and the necessity of designing systems for failure, which involves planning for system errors and ensuring smooth recovery. Shetty shares her approach of using a "carrot-and-stick" method to emphasize the benefits of proactive monitoring, while also acknowledging the ongoing challenges of integrating diverse monitoring tools. The discussion underscores the need for continuous evolution in monitoring practices, akin to software development cycles, and suggests practical exercises like incident drills to prepare teams for real-world system failures.

Matt Stewart shares his experience of using Grafana Cloud to monitor his Tesla Model 3 by leveraging Tesla APIs and a Teslafi account. He outlines the process of setting up a Raspberry Pi with Prometheus and Grafana Cloud to visualize various car metrics such as charge state, miles driven, and OTA updates. Through a detailed step-by-step guide, he explains how to install necessary software on the Raspberry Pi, configure Prometheus to collect and export data, and use teslafi_exporter to gather data from the Tesla APIs. The setup enables users to monitor Tesla data within Grafana's dashboard, showcasing the flexibility and capability of Grafana Cloud for diverse monitoring applications.