March 2021 Summaries

Exemplars are revolutionizing distributed tracing in the observability landscape by optimizing costs and enhancing performance, similar to the disruptions caused by Prometheus and Grafana Loki in metrics and logging, respectively. Exemplars function by linking significant metrics and logs to traces via an ID, offering a streamlined path for analyzing interesting traces without the need to search through extensive data. Grafana Tempo, an open-source distributed tracing backend by Grafana Labs, capitalizes on this innovation, allowing for seamless integration of metrics, logs, and traces at a massive scale without sampling. This holistic approach, supported by the OpenMetrics specification, leverages existing indexes to save on costs and time, transforming how traces are stored and retrieved. This advancement reflects years of vision and preparation in observability, with Grafana Labs playing a pivotal role in its development.

Grafana Labs has introduced a new Jira Enterprise plugin designed to integrate Jira data with Grafana's visualization capabilities, offering comprehensive insights into software development processes. The plugin allows users to visualize Jira issues, such as bugs and user stories, alongside metrics from other data sources, facilitating a more refined planning process and enabling teams to improve sprint velocity, issue throughput, and mean time to resolution. Users can install the plugin, input their Atlassian instance URL, email, and API token, and begin querying Jira issues using the Jira Query Editor and JQL filters to narrow search results. The plugin also supports transformations to perform complex calculations, offering insights like completed versus estimated story points and progress against epics. Additional features include dynamic filtering with template variables, annotations for correlating Jira events with other metrics, and alerts for monitoring sprint estimates. The Jira plugin is available for Grafana Cloud and Enterprise license users, and the blog post encourages feedback to improve the plugin further.

Grafana Labs has been experimenting with its Grafana Agent to enhance the Prometheus ecosystem, focusing on creating a single Agent for its observability stack. Initially launched in March 2020, the Grafana Agent aimed to streamline metrics by removing TSDB storage, and over time, it introduced support for logs and traces, evolving into a comprehensive tool that integrates various projects. The team has tested several features, such as host filtering and scraping service modes, addressing challenges like service discovery in large clusters and load distribution. Successful experiments like the remote write-only Prometheus and SigV4 support have been upstreamed, with the Prometheus team integrating these functionalities into the official project. By maintaining a smaller user base, Grafana Labs can experiment extensively and refine features before contributing them to Prometheus, benefiting both Grafana and Prometheus users with stable and innovative solutions. The ongoing collaboration and contributions to Prometheus highlight a strategic approach to evolving open-source observability tools.

On March 11, a two-hour outage occurred in the Grafana Cloud Hosted Prometheus service in the US-central region due to a new tenant exceeding data limits, which a bug failed to enforce properly, causing a cascading failure across the system. The issue arose after a tenant, unaware of their excessive data transmission, overwhelmed a cluster despite preconfigured limits meant to prevent such occurrences, leading to overloaded internal paths and failures in authentication gateways and load balancers. Grafana's response included scaling up the cluster, imposing stricter tenant limits, and fixing the bug by March 12, which restored service. To prevent future incidents, they reverted the change in limit handling, introduced new protective limits, improved per-tenant limit management, and enhanced cluster scaling and provisioning processes, alongside investing in incident response training and automation. These measures aim to prevent individual customers from overwhelming clusters and reduce future recovery times.

Grafana 7.5 introduces several notable enhancements, including the new pie chart panel, which is now a fully integrated visualization option, and alerting support for Loki, allowing users to create alerts on Loki metric queries. The release also features a more user-friendly Logs browser for navigating Loki labels, and Grafana Tempo has been added as a backend data source for improved trace data handling. Elasticsearch users benefit from word highlighting in log searches, and PostgreSQL setup is simplified by allowing SSL certificates to be pasted directly into the UI. Cloudwatch data source improvements include support for additional AWS metrics, region-specific enhancements, and configuration options to reduce latency. Grafana Enterprise users gain query caching for faster dashboard loads and support for template variables in reports, while active user limits are enforced based on license terms. The update encourages users to explore these features through detailed documentation, changelogs, and available webinars, emphasizing community contributions in the development process.

In 2021, Goutham Veeramachaneni, a maintainer for Cortex, discussed the potential for Cortex to become a viable database solution for Internet of Things (IoT) data. Originally developed as a scalable, long-term storage solution for Prometheus, Cortex is built around a time series database (TSDB) that offers high scalability and cost efficiency. Despite its strengths, Cortex's current protocol limitations hinder its suitability for IoT, prompting the creation of influx2cortex, a write proxy to support various protocols. Veeramachaneni highlights the importance of addressing out-of-order data handling, which remains a challenge, and expresses interest in expanding Cortex's use cases beyond just Prometheus long-term storage to include IoT applications. The aim is to refine Cortex's capabilities by experimenting with IoT homelabs and engaging with users, ultimately hoping to attract both hobbyists and serious customers to use Grafana Cloud for IoT data storage and analysis.

Dave Schmid, a Senior Solutions Engineer at Grafana Labs, shares his newfound appreciation for logs, particularly through the use of Grafana Loki, as he resolved Wi-Fi issues in his home network. Initially a metrics enthusiast, Schmid transitioned to logging when metrics alone couldn't solve his connectivity challenges. He recounts how Loki, alongside Prometheus and Kubernetes, provided a robust solution for aggregating and exploring logs. By deploying Loki and its components like Promtail and syslog-ng, he successfully monitored his network devices and transformed logs into metrics using LogQL queries, ultimately identifying the root cause of his network problems. Schmid's experience underscores the ease of connecting open-source solutions to address technical issues and is detailed in his "Loki Syslog AIO" project, available on GitHub for others to replicate.

In March 2021, Grafana Cloud introduced several updates to enhance user experience, focusing on improvements in alerting, synthetic monitoring, and system integrations. Users will notice easier installation and management of agents and integrations, now bundled for various operating systems, with simpler setup and uninstallation processes. The cloud alerting interface has been refined for better readability, featuring collapsible rule groups and improved time range selection for scheduling silences. Synthetic monitoring now offers a redesigned checks page for better usability, allowing users to manage checks directly from the main page with enhanced sorting and filtering options. Additionally, new guides have been released to assist users in integrating their existing logs and Kubernetes metrics with Grafana Cloud.

Vicky Lee, a software engineer at Grafana Labs, specializes in developing Enterprise plugins, focusing on premium, paid-for plugins due to her interest in the observability space and aspirations for engineering management. Originally from Yorkshire and now based in London, she enjoys a time zone advantage over her North American colleagues. In her free time, she engages in virtual karaoke, gaming, and binge-watching TV shows, with a particular fondness for memes on Tumblr and TikTok. Vicky values her Roomba and Braava jet for simplifying household chores and prefers listening to a diverse range of music while coding. Her work-from-home tips include keeping pets out of the room during calls and checking open tabs before screen sharing.

Utilita, the U.K.'s first specialist Smart Pay As You Go Energy supplier, has effectively leveraged Grafana Enterprise to enhance operational efficiency and customer experience. Since adopting Grafana in 2019, Utilita has utilized its enterprise features and plugins, particularly the Oracle plugin, to streamline processes such as sales, payment, and customer support monitoring, leading to significant time and cost savings. The implementation of real-time dashboards has addressed previous issues, such as identifying ineligible customers before sign-up and monitoring incomplete payment processes, thereby reducing resource wastage. Additionally, Grafana's alerting capabilities have enabled proactive responses to power outages and improved customer interactions. Utilita is further innovating by developing features like predictive appliance usage, which is currently in beta, aimed at providing customers with tailored energy efficiency advice. This strategic use of Grafana has positioned Utilita to quickly and cost-effectively integrate new digital technologies into their business model.

GrafanaCONline is set to return from June 7-17, offering a virtual experience for the global Grafana community after last year's successful pivot from an in-person event due to the pandemic. This online event will feature over 20 live sessions, including demos and workshops, showcasing the upcoming Grafana 8.0 release and engaging the community with stories and presentations. The call for papers (CFP) is open, inviting submissions for 20- and 40-minute talks on a variety of topics such as IT observability, health and sciences, IoT, business intelligence dashboards, and homelab projects, with a particular encouragement for first-time speakers and underrepresented groups to participate. The deadline for submissions is April 9.

Grafana Labs has introduced new features in Grafana Enterprise Metrics (GEM) versions 1.1 and 1.2, enhancing its scalable, self-hosted Prometheus service. Key additions include label-based access control, which allows users to create access policies based on labels associated with their Prometheus metrics, addressing real-world challenges of access management. Another feature is instance federation, enabling cross-instance queries, which facilitates data visualization across multiple teams or departments while maintaining access restrictions. Grafana Labs has also developed a scalable compactor capable of handling up to 650 million active series, significantly improving upon the previous limitations of the Cortex architecture. These enhancements aim to offer users greater control, scalability, and flexibility in managing their metrics infrastructure.

Uche Obasi, a software engineer at Grafana Labs based in Nigeria, focuses on developing the frontend of Grafana, attracted by the company's open-source culture, remote-first environment, and growth opportunities. Before joining Grafana Labs, Uche contributed to the Thanos project through the LFX Mentorship program, marking his entry into open-source contributions. An active member of the CNCF community, Uche also organizes events, such as the upcoming Kubernetes Community Days Africa. In his free time, he enjoys learning basketball, socializing with friends and family, and writing. To maintain focus at work, Uche turns off social media notifications and listens to lo-fi music. He finds satisfaction in enhancing user experience through his work and appreciates his talented colleagues. Uche's favorite dish is party jollof rice, a popular Nigerian meal, and he looks forward to attending in-person developer conferences post-COVID.

Grafana Loki 2.2 introduces significant enhancements, including multi-line log support in Promtail, crash resiliency through a write-ahead log (WAL) for Loki's ingester, and numerous performance improvements. The release, which resulted from over 200 merged pull requests, also features optimizations in JSON parsing and query paths, reducing core usage and memory consumption. Notable contributions include enhanced JSON element selection and Google Pub/Sub integration for log syncing, as well as Windows event log support in Promtail. These updates aim to improve log reliability and efficiency, with Grafana Cloud offering a free plan for new users to explore Loki's capabilities.

The text discusses the practical and economic advantages of the Internet of Things (IoT) and its compatibility with Prometheus for data management. IoT involves deploying affordable, internet-connected sensors with computing capabilities, such as ESP32 or Raspberry RP2040, in various locations, allowing for extensive data collection. These sensors can operate in diverse environments, thanks to wireless technologies like WiFi, Zigbee, and LoRaWAN, and do not require centralized control, enabling decentralized decision-making. Prometheus is highlighted as an effective tool for managing IoT data, as it can efficiently handle time series data from multiple sources, including weather data, through its time series database (TSDB) design and PromQL's mathematical capabilities. The text hints at future discussions on related topics by Goutham Veeramachaneni, emphasizing Prometheus's foundational roots in weather data analysis.

Grafana Enterprise Logs (GEL) enhances the Grafana Enterprise Stack by introducing a centralized tenant administration feature through its admin API and plugin, building upon the log aggregation capabilities of Grafana Loki. The system supports multiple tenants, allowing for tailored data management through tenant-specific IDs and offering centralized control over data access and policies. Administrators can create tenants, set access policies, and generate tokens either via the admin API, which integrates with existing provisioning systems like Puppet or Chef, or through the Grafana Enterprise Logs administration plugin for a user-friendly interface. These features facilitate the secure and efficient management of log data, enabling organizations to restrict or control access according to their needs. The system also allows for flexible integration with Grafana for data visualization and analysis, supporting best practices in log management.

Ivana Huckova details her experience building a monitoring system for her avocado plant using Arduino and Grafana Cloud during a Grafana hack day. Inspired by her previous sourdough monitoring project, she used an ESP32 development board and various sensors to measure parameters like plant height, air temperature, humidity, soil moisture, and light conditions. Huckova utilized the Arduino IDE for programming and connected the hardware to Grafana Cloud to visualize the data in real-time dashboards. She explains how to set up the necessary software and hardware, shares her Arduino sketches, and provides guidance on creating similar IoT monitoring projects. Through this project, she aims to demonstrate that building a custom monitoring system can be both accessible and enjoyable.

Grafana Labs showcases team member Michaela Guzzi, a recruiter specializing in building the company's Go-to-Market team, who is based in Denver, Colorado. Her decision to join Grafana Labs was influenced by the company's passionate team, strong mission and values, impressive growth, and status as a remote-first pioneer, which she finds inspiring to discuss with potential candidates. Michaela appreciates the diversity and daily interactions with people from over 37 countries, facilitated by the company's fully remote structure. Outside of work, she enjoys outdoor activities like skiing and camping, cooking with her grandmother's pasta maker, reading historical fiction, and recently fulfilled her dream of moving from New York to Colorado.

Grafana Labs is partnering with Elastic to develop an official Elasticsearch plugin for Grafana, reflecting their "big tent" philosophy that emphasizes user choice and interoperability in observability strategies. This collaboration aims to enhance the existing Elasticsearch plugin, allowing Grafana users to visualize data from one of the most popular platforms alongside multiple other data sources. The partnership will leverage the combined engineering expertise of Grafana Labs and Elastic to provide a seamless experience, expanding on Grafana's origins as a project derived from Kibana, the Elasticsearch platform's frontend. The integration will be available to all Grafana users, and the development will focus on adding features like more aggregations, broader query language support, and space-saving constructs, with community feedback playing a key role in its evolution.

Setting up a Grafana development environment on a Windows PC using Windows Subsystem for Linux (WSL) involves several steps to ensure compatibility with the Linux-based system that Grafana requires. This process includes installing WSL and a suitable Linux distribution such as Ubuntu v18.04, along with necessary development dependencies like Go, Node.js, Yarn, and Docker, which requires WSL2 for successful operation. Grafana's code repository should be cloned using Git, and the application comprises both frontend and backend components that need to be run concurrently. The frontend requires specific dependencies installed using Yarn, while the backend is initiated via a make run command. Developers can also add data sources through a script in the devenv directory, and the whole setup allows contributors to explore and contribute further to Grafana's development. The setup guide emphasizes using the Linux environment for all installations to avoid issues that arise from running concurrent Windows and Linux environments.

Grafana 7.4 introduces the ability for Grafana Enterprise customers to export raw usage insights to Loki, enhancing the management of dashboard sprawl and improving troubleshooting efficiency. Previously, users could visualize some usage insights, but lacked access to raw data, limiting their customization options. With this update, administrators can now log details of each dashboard visit and data source query, allowing for the creation of detailed dashboards that aid in faster problem resolution and user support. The integration with Loki facilitates ad hoc queries, dashboard building without external tools, and the identification of adept users within a network. This feature aligns with the launch of Grafana Enterprise Logs, which offers a secure, scalable solution for log indexing and storage, enhancing companies' ability to manage permissions and access in compliance with security policies.

A new monthly webinar series by Grafana Labs is launching on March 4, featuring 30-minute live demos led by their solutions engineering team. The first session, hosted by Vijay Tolani and John O’Brien, will guide participants on how to utilize the Grafana Stack to achieve observability quickly, covering aspects like setting up monitoring for web services using Prometheus and Loki, exploring and correlating data within Grafana, and setting up alerts. Participants are encouraged to suggest topics for future webinars, and Grafana Labs highlights Grafana Cloud as an accessible entry point for using Grafana, Prometheus, Loki, and Tempo, with a new free plan and enhanced paid plans available.