September 2022 Summaries

"Grafana's Big Tent" podcast, hosted by Mat Ryer, Matt Toback, and Tom Wilkie, explores the world of observability through engaging discussions with various guests, covering topics like logging, service level objectives (SLOs), and technical debt. The hosts reflect on memorable episodes from the first season, such as discussions on the human readability of logs, the complexities of SLO negotiations, and the importance of being kind to one's future self through smart software practices. The podcast highlights the balance between avoiding excessive technical and organizational debt while allowing for flexibility and experimentation. The hosts also emphasize the need for an "error budget" in life and podcasting, encouraging risk-taking and learning from failures. As they plan for the second season, they invite listener feedback and ideas, underscoring the community-driven nature of the podcast.

eBPF, or extended Berkeley Packet Filter, leverages the kernel's capabilities to execute sandboxed programs within a privileged context, making it useful for system-wide profiling by collecting stack traces at regular intervals. The intersection of eBPF and profiling has seen significant growth, with projects like Pyroscope, Pixie, and Parca gaining traction for their contributions to observability. Profilers can operate at the user-space level, suitable for native-language profiling, or at the kernel level through eBPF, each with distinct pros and cons. User-space profilers allow detailed tagging and simplicity in local development but face challenges in multi-language systems, while eBPF profilers offer fleet-wide metrics and infrastructure-level insights but require specific Linux kernel versions and have limitations in user-level code tagging. Pyroscope proposes a solution that combines the strengths of both eBPF and native-language profiling to maximize profiling value. This dual approach is particularly useful for profiling compiled languages, though eBPF faces challenges with interpreted languages. Pyroscope's integration with Kubernetes allows for detailed CPU usage breakdowns, and future efforts aim to further merge kernel and user-space profiling for enhanced observability.

Grafana Agent Flow is an experimental feature introduced in the Grafana Agent v0.28.0 release, designed to provide a more intuitive and powerful way to configure, run, and debug Grafana Agent. This new feature aims to simplify the learning curve for new users while empowering experienced users to create complex workflows using programmable pipelines composed of small, reusable components. Built on River, a Terraform-inspired declarative language, Flow allows dynamic configuration through expressions and utility functions, enabling components to emit output values that bind together in a pipeline. This event-driven evaluation system facilitates real-time updates and immediate feedback on configuration changes, enhancing the user experience beyond the limitations of the static Agent mode. Currently focused on Prometheus metrics pipelines, Flow is set to expand with support for logs, traces, and integration with OpenTelemetry Collector components, promising a more robust future for Grafana Agent users.

Grafana Labs transitioned from using Consul to embedded memberlist to manage distributed databases, focusing on improving scalability and reliability. The migration involved configuring a multi-KV store client to initially use both Consul and memberlist, gradually shifting all traffic to memberlist to eliminate Consul as a single point of failure. This change was prompted by limitations in Consul's performance under heavy loads, primarily due to its compare-and-swap operations and additional read requests. By adopting memberlist, which utilizes a gossip protocol, Grafana Labs reduced network traffic and removed the dependency on a dedicated KV store, allowing for more efficient scaling of their components. The migration process included careful management of Kubernetes labels to ensure that memberlist communications were properly segregated across namespaces, preventing cross-namespace interference and potential data loss. This strategic shift not only enhanced the scalability of their production environments but also allowed for more resilient and efficient deployment models. The Grafana Mimir team's contributions were instrumental in developing the internal memberlist libraries and migration steps, which were adapted for use by the Loki team.

In a recent project for the European Space Agency, Armin Müller from ScopeSET explored ways to expedite the training of machine learning models by integrating Grafana into their workflow. The project involved analyzing time series data from satellite sensors to support automated anomaly detection, and the data was initially stored in a PostgreSQL database as part of the CubeSAT MOVE-II demo mission. By transitioning from slower CSV visualization tools to Grafana, the team significantly reduced the time needed to process and visualize large datasets, enhancing their ability to identify nominal data ranges for model training. This integration allowed for a more efficient analysis process, as Grafana's performance and UI responsiveness excelled with over 100,000 rows of data, compared to previous tools. The use of Grafana, combined with JupyterLab and Python, streamlined the anomaly detection workflow, offering a multiplier effect in efficiency across various machine learning models due to its capacity to handle diverse anomaly scenarios.

Emil A. Siemes and Stefan Kupstaitis-Dunkler, Solutions Engineers at Grafana Labs, describe a project integrating Garmin devices with Grafana Cloud to analyze fitness metrics such as heart rate. By using Garmin's CONNECT IQ SDK and Grafana Loki, they demonstrate how to collect and visualize data from IoT sensors like sport watches, which sync through smartphones acting as edge devices. This architecture showcases a broader IoT application where simple devices without direct internet access communicate with more powerful edge devices. The process includes setting up a stateless and scalable infrastructure using Google Cloud Run to manage data flow into Grafana Cloud Logs, where users can create dashboards to visualize running metrics. This integration not only serves fitness enthusiasts but also offers a model for IoT systems in various industries.

NOS, a Portuguese telecommunications and media company, has significantly enhanced its network monitoring capabilities by adopting Grafana, an open-source observability tool, since 2017. Originally relying on the Business Objects suite for static reports, NOS faced challenges with limited data filtering and inefficient analysis, prompting a shift to Grafana for more dynamic and comprehensive data analysis. With Grafana, NOS engineers can troubleshoot issues in less than a minute, allowing them to efficiently optimize the network and address customer problems. The tool’s ease of use has led to widespread adoption across various teams, enabling them to create custom dashboards and explore new use cases, thus providing a holistic view of the network and improving overall connectivity. As a result, NOS has achieved greater efficiency and better service delivery, ultimately enhancing customer satisfaction in the competitive telecommunications industry.

Yonatan Mevorach, an Infrastructure Team Lead at Wix, creatively repurposed a mini-arcade machine into a Grafana dashboard display using a Raspberry Pi, addressing the challenge of remote work during COVID-19 that limited him to a single monitor. Motivated by the need to monitor a newly launched service more closely, Mevorach reverse-engineered the arcade machine, replacing its components like the CPU with a Raspberry Pi Zero, which allowed for programmability, internet connectivity, and continuous power via USB. By examining the circuit board and researching online, he identified the necessary connections, including the ground pin, and swapped in an LCD screen compatible with the Raspberry Pi. Despite his background in JavaScript, he adapted the necessary Python code to JavaScript to operate the display drivers effectively. After successfully rendering Grafana graphs on the screen using the Grafana API, Mevorach completed his project by soldering the components together, marking his first venture into electronics hacking and expressing interest in future projects that might be battery or solar-powered.

ObservabilityCON 2022, organized by Grafana Labs, is set to take place on November 2 in New York City, offering attendees insights into the latest developments in open source observability tools and strategies. The event features a keynote by Grafana Labs' CEO Raj Dutt, VP of Technology Tom Wilkie, and the engineering team, focusing on advancements in the LGTM observability stack, along with exciting announcements. With a limited capacity of 200 seats, early-bird tickets are available at a discounted rate until October 21. The conference agenda includes sessions on scaling observability, performance testing, and incident response, along with success stories from companies like Wells Fargo, Adobe, and Banco Itaú that have leveraged Grafana's tools. Additionally, expo booths will provide opportunities for attendees to interact with technical experts and explore demos, while pre-conference workshops on November 1 offer hands-on guidance in four specialized tracks. An observability survey is also being conducted to gather insights from participants, with results to be shared during the event.

Grafana has released version 9.1.6, addressing moderate severity security vulnerabilities CVE-2022-35957 and CVE-2022-36062, which were related to privilege escalation in Grafana Auth Proxy and role-based access control (RBAC). These vulnerabilities could lead to unauthorized access permissions, with CVE-2022-36062 allowing Editors and Viewers to access Admin-only folders or dashboards, and CVE-2022-35957 enabling escalation from Admin to Server Admin through Auth proxy. The fixes are included in patches for Grafana versions 9.0.9 and 8.5.13, with updates also applied to Grafana Cloud. Users are advised to upgrade their instances or apply alternative mitigations if upgrades are not feasible. Security issues can be reported to Grafana, with information available on their security blog and RSS feed.

The complex nature of alerting infrastructure requires scalable solutions, which are often challenging to manage manually across different teams and organizations. By utilizing Infrastructure as Code (IaC), such as the recently released Terraform provider support for Grafana Alerting, organizations can streamline their alert management processes by defining observability tooling as code. This enables storing configurations in version control and deploying them alongside other infrastructure components. The integration allows users to create, manage, and maintain their entire Grafana Alerting stack as code, simplifying the process of keeping alerts in sync with code changes. Key functionalities include creating contact points for notifications, such as Slack, setting up notification policies and routing, defining mute timings, and establishing alert rules against any Grafana data source. The new Terraform support provides a structured approach to managing alerting infrastructure, enhancing efficiency, reducing errors, and ensuring consistency in notifications.

This informative piece outlines the process of configuring Grafana Loki and its client Promtail to seamlessly receive logs from applications hosted on Heroku without the need for complex scripts. The guide begins by explaining Heroku's logging system, which utilizes HTTP drains to route logs from applications via LogPlex to endpoints that understand Heroku’s log format. Promtail, now equipped with a specific target for Heroku, can act as a drain to streamline this log shipping process. The text provides detailed instructions on setting up a Promtail instance on Heroku, including creating necessary configuration files and environment variables, and deploying the application using Heroku's Git model. It emphasizes the importance of using Grafana's API keys and job names in the setup, and concludes with steps to verify that logs from a Heroku application, referred to as RealApp, are successfully being sent to a Grafana Loki instance. The setup, while demonstrated using Grafana Cloud, is adaptable to other environments as long as the Promtail instance is publicly accessible for Heroku's log drains.

Metrics play a crucial role in observability by providing insights into a system's performance, and Grafana Cloud offers a managed service to streamline the process of monitoring these metrics. The decision on how to best ship metrics to Grafana Cloud depends on several factors, such as the source of data, familiarity with observability tools, and the scale of metrics to be monitored. Grafana Cloud supports multiple methods for shipping metrics, including HTTP Write, Agent, and Prometheus Remote Write, each suited to different user needs and technical expertise. The guide emphasizes the importance of monitoring "The Four Golden Signals"—latency, traffic, errors, and saturation—to gain meaningful insights into system performance. Once metrics are shipped to Grafana Cloud, users can leverage the Explore tool to create and visualize queries, facilitating real-time monitoring and data analysis. The platform simplifies the management of metrics by handling scaling, security, and updates, allowing users to focus on creating and managing applications without the overhead of maintaining a metrics database.

Grafana Labs has developed a tool called Levitate to assist developers in checking the compatibility of their plugins with various versions of the Grafana plugins API. This tool, designed to detect breaking changes quickly, can be integrated into a CI pipeline or used as a CLI on a local machine, allowing developers to assess if their plugins will function correctly with both older and updated versions of Grafana. While not aiming for complete error elimination, Levitate aims to boost developers' confidence by providing insights into changes in Grafana, facilitating quicker adaptation of code. The tool is easy to use with commands specifying compatibility checks against specific Grafana versions. Developers can integrate Levitate into their workflows and contribute feedback on its utility. Additionally, Grafana Labs offers resources such as webinars to explore its plugin ecosystem and a free trial of Grafana Cloud Pro for enhanced observability.

Grafana Incident, now generally available to all Grafana Cloud users, is designed to streamline incident management by automating administrative tasks to reduce mean time to repair (MTTR) and allowing teams to concentrate on resolving issues. Initially launched in preview in February, the tool has benefited from user feedback and is now equipped with features such as Slack integration, automated Google Docs templating, and StatusPage integration, which enhance its utility for incident response. Grafana Incident facilitates coordination and communication during incidents by setting up necessary channels and meeting rooms, assigning incident roles, and providing a chatbot for managing incidents. It also includes a Suggestbot feature that uses machine learning to recommend relevant dashboards and a task manager to track to-do items. Additionally, the tool automatically creates an activity timeline and offers a Present tool for postmortem reviews, aiding in transparency and future incident prevention. Existing Grafana Cloud users can access Grafana Incident through the platform, while new users can start with a free account or trial for more comprehensive plans.

At the Cisco Live event, a premier annual gathering in the networking industry, Jason Davis and his team used Grafana dashboards to effectively visualize and monitor real-time data for 28,000 attendees, showcasing metrics like CPU usage, signal strength, and Internet traffic in engaging formats. The event required a robust IT infrastructure with 2,300 wireless access points and 650 network switches to ensure seamless connectivity across multiple venues. Upgrading from Grafana 6.1 to 8.4 for the 2022 event, the team utilized new visualizations and features, such as ’90s retro LED bar graphs and radar graphs, to create impactful dashboards that appealed to both technical and non-technical audiences. These dashboards included playful visualizations, like comparing Internet data to Marvel movie volumes, making complex data accessible and entertaining. By migrating to Grafana Cloud and leveraging duplicate data pushes, Davis's team explored the potential of cloud-based management for future events, continuing to enhance the attendee experience with innovative and dynamic visual displays.

Paul Leroy, an experienced IT consultant, and his daughter Astri embarked on a project to monitor their pet python, Pretzel, using a range of technologies including Grafana, InfluxDB, Mosquitto, and Node-RED. The monitoring system, developed to ensure the python's vivarium maintains appropriate temperature and humidity levels, utilizes sensors and a Kubernetes cluster running on a Raspberry Pi setup. Astri contributed significantly by constructing the vivarium and soldering sensors. The data from the sensors is processed and visualized on a Grafana dashboard, while alerts are managed through Slack and a mobile UI for remote control. Despite its success in keeping Pretzel healthy, Paul plans to improve the system by enhancing alert mechanisms and testing alternative communication technologies to replace the current Wi-Fi setup. This project not only highlights the practical application of IoT technologies in pet care but also serves as a learning experience in integrating various tech platforms for a domestic setting.

Podman, a daemonless container engine alternative to Docker, facilitates the deployment of the Grafana stack, which includes Loki, Grafana, Tempo, and Mimir, by using OCI-compliant container images on Linux, Mac, and Windows operating systems. The blog post explains how to set up the Grafana stack using Podman on a Linux host, starting with updating system packages and installing Podman. The deployment process involves cloning a GitHub repository containing a Kubernetes-style YAML configuration file and using Podman's compatibility with Docker commands and Kubernetes pods to set up the containers. Once the containers are running, the services can be exposed using firewall rules, and users can log into Grafana, add data sources for metrics, logs, and traces, and begin using the stack for local development or proof of concept projects. The process demonstrates the simplicity and flexibility of deploying a comprehensive observability solution with Podman, offering benefits for both open-source and Grafana Enterprise users.

Grafana Mimir 2.2 introduces experimental support for out-of-order sample ingestion, addressing the traditional limitation of Prometheus TSDB, which only accepts in-order samples. This feature caters to use cases like IoT devices, message buses, and isolated Prometheus instances that require the ingestion of older metrics. The design involves storing the last 30 out-of-order samples in memory, utilizing a Write-Behind-Log (WBL) to record ingested samples, and compacting these samples into persistent blocks. Out-of-order samples are managed using a new configuration parameter, out_of_order_time_window, allowing users to specify the permissible age of these samples. This capability is being tested in Grafana Cloud and will be enabled for all users after further testing. The feature is developed in the grafana/mimir-prometheus GitHub repository, allowing for experimentation and problem-solving before contributing upstream. While the feature shows varied performance impacts, including increased CPU usage during high rates of out-of-order sample processing, memory usage remains stable. This advancement positions Grafana Mimir as a versatile time series database capable of integrating with various metrics systems like Graphite, Datadog, Influx, and Prometheus.

Myrle Krantz, Senior Engineering Manager at Grafana Labs, emphasizes a management style that fosters autonomy, responsibility, and team collaboration within her globally distributed Application Observability team. Rather than micromanaging, she prioritizes regular check-ins to provide necessary context and tools, allowing team members to be self-starters, which she believes most people prefer. Myrle values candidates who take ownership, learn from mistakes, and collaborate beyond their defined roles, highlighting the importance of social and psychological safety in creative work. Her team, which spans across various time zones, stays connected through weekly meetings, daily asynchronous updates on Slack, and optional social gatherings, promoting a sense of community and shared purpose. Myrle's approach involves looking for engineers who not only possess technical skills but also demonstrate a proactive attitude and willingness to engage deeply with challenges, ensuring they contribute to and grow within the team.

Houzz, a leading platform for home remodeling and design, undertook a significant upgrade of its tech stack to address scalability challenges and enhance operational efficiency. The company transitioned from monoliths to microservices and revamped their observability tools by adopting Grafana Loki for log visibility and Grafana Tempo for tracing. This shift aimed to streamline log searches and improve network observability amidst growing data volumes. By implementing these tools, Houzz reduced Mean Time to Resolution (MTTR) and Mean Time to Identify (MTTI), leading to increased uptime and improved developer and customer satisfaction. The integration of Grafana's tools allows Houzz to efficiently correlate metrics, logs, and traces, providing a unified view of system performance and issues, thereby enhancing their ability to diagnose and resolve problems quickly.

The new Grafana Ansible collection is designed for Grafana Cloud users, offering a streamlined approach to manage various resources such as plugins, dashboards, and API keys through Infrastructure as Code practices. Available on Ansible Galaxy, the collection includes eight modules specifically for Grafana Cloud management, with plans to expand further. Users can install the collection using simple commands and integrate it into their existing Ansible workflows. The initiative encourages community involvement through bug reporting, feature requests, and code contributions on GitHub. Grafana Cloud also offers a free trial for new users, promoting its comprehensive observability features.