June 2021 Summaries

Google Cloud Monitoring's data source plugin for Grafana has received several updates aimed at enhancing user experience and functionality. The collaboration between Google and Grafana Labs has led to the integration of Google Cloud Monitoring as a built-in data source within Grafana, simplifying the creation of dashboards. Recent improvements include the addition of 15 popular sample dashboards from the Google Cloud Monitoring dashboard samples library, allowing users to easily import, edit, and apply them. Another enhancement is the introduction of deep linking, which facilitates seamless navigation between Grafana and the Google Cloud console, specifically targeting the desired time series data for troubleshooting. Additionally, the plugin now supports an improved query interface that aligns with Cloud Monitoring's new dashboard creation flow, enabling advanced visualization options for delta and cumulative metrics. Furthermore, the Monitoring Query Language (MQL) editor has been integrated, allowing users to execute MQL queries directly from Grafana. These updates are designed to streamline the process of building and managing dashboards for Google Cloud Monitoring metrics, furthering the collaboration's goal of offering robust analytical tools within Grafana.

American Metal Processing (AMP), a 75-year-old heat treating company, significantly enhanced its manufacturing efficiency by adopting Grafana Cloud, a fully managed platform that simplifies data collection and visualization. During the GrafanaCONline talk, President Grant Pinkos explained how the platform helps AMP minimize unplanned downtime by identifying patterns and predicting potential failures before they occur. At AMP's large rotary heat treat facility, sensor data is collected and stored in databases like Influx and Microsoft SQL, then displayed through Grafana Cloud dashboards accessible throughout the plant and on mobile devices. This system provides real-time observability, allowing the small workforce of 18 employees to respond promptly to issues, even outside regular working hours. The integration of Grafana Cloud replaced traditional large panels with flexible dashboards, enabling continuous monitoring and improvement of processes by visualizing problems and detecting trends as they happen.

Grafana 8.0 introduces significant updates tailored for industrial and IoT applications, with a focus on real-time data streaming to dashboards via Grafana Live. This core feature allows events and data to be broadcast in real-time, overcoming previous limitations in Grafana 7 where streaming required custom plugins and data sources. Now, data can be streamed directly out of the box, facilitating new applications and use cases. The update supports real-time dashboard events, such as live changes in visualizations and sensor data streaming through plugins like MQTT or direct server updates via HTTP and WebSocket endpoints. Demonstrations highlighted the system's ability to handle data from accelerometers, illustrating the platform's enhanced integration capabilities with tools such as Telegraf. Future updates, including the option to stream to Grafana Cloud in version 8.1 and beyond, promise to expand Grafana's functionality further for IoT and industrial users.

Siemens employs IoT sensor data and Grafana to enhance train maintenance and operations, as described by Martin Klimmek and Haluk Tutuk during their GrafanaCONline 2021 talk. With a team of data scientists, engineers, and a network of over 3,000 suppliers, Siemens prioritizes observability to better understand and control assets. Grafana enables real-time data generation from trains, optimizing maintenance and supporting complex tasks like HVAC system management and passenger capacity counting, which proved valuable during the pandemic for social distancing. Siemens pushes technical and collaborative boundaries to create a platform that facilitates expertise sharing, experimentation, and rapid feature development. The approach integrates millions of data points from 500 UK trains daily, promoting a passenger-centric operation and fostering a collaborative environment that enhances observability and predictive capabilities.

Grafana's vibrant community celebrated the conclusion of GrafanCONline and the launch of Grafana 8 and Tempo 1.0 by showcasing a series of community-built dashboards, demonstrating the platform's versatility across various applications. These 12 dashboards, available for easy integration into any Grafana instance, cater to diverse monitoring needs such as global traffic, VMware vSphere, Valheim server performance, GitHub repositories, Splunk Infrastructure Monitoring, Jitsi Meet conferences, MinIO data infrastructure, home energy usage, and more. Notably, these dashboards leverage various data sources like InfluxDB, Prometheus, and specific plugins to provide insightful visualizations and metrics. The showcase emphasizes Grafana's utility in monitoring and analyzing data from different systems, encouraging users to explore and customize dashboards to fit their specific needs. Additionally, users are invited to enhance their dashboard design skills through a free webinar, underscoring Grafana's commitment to community engagement and education.

GrafanaCONline Day 6 covered a variety of topics surrounding Grafana's functionalities and business developments, featuring discussions on Loki for logs, Grafana's role in high-performance computing, and Salesforce's use of Grafana for service health management. Danny Kopping and Cyril Tovena highlighted Loki's integration into Grafana and its benefits, such as low cost and familiar query language, while Vincent Behar explained Jenkins X's use of Grafana for visualizing Continuous Delivery Indicators. J.D. Maloney from NSCA shared how Grafana aids in managing HPC infrastructure, and Salesforce representatives detailed their use of Grafana for observability and service health insights. The conference concluded with Grafana Labs CEO Raj Dutt discussing the company's growth, the acquisition of load-testing startup k6, and the future enhancements expected in the Grafana Stack. All sessions are available on demand, allowing participants to revisit the insights shared during the event.

GrafanaCONline 2021 Day 5 featured several sessions that highlighted the latest advancements in Grafana's capabilities, including a unified alerting system in Grafana 8.0 that standardizes alert behavior across all its platforms and introduces features like multi-dimensional alerting. The event also showcased improvements in "as code" practices for Grafana dashboards, enabling easier installation and upgrades through tools like GitHub. Yashvardhan Kukreja discussed using Grafana Cloud's synthetic monitoring to enhance user experience tracking, while Dénes Csala presented a customized Grafana dashboard for monitoring COVID-19's economic impact in Romania. Ronan Higgins described Grafana's role in managing data from NASA's SOFIA observatory, and Boris Gdalevich demonstrated KKR's 360-degree observability platform using Grafana Enterprise tools. These sessions and others emphasized Grafana's integration into diverse fields, from astronomy to financial services, and will be available for on-demand viewing.

Daniel Edgar's blog post discusses the integration of Grafana and the NGINX Instance Manager to monitor the health of NGINX instances, emphasizing that the content pertains to an older version of the NGINX Instance Manager, with newer versions offering different connection methods. Grafana is highlighted as a robust observability platform capable of generating operational insights by querying Prometheus endpoints, which is complemented by the NGINX Instance Manager's ability to manage and collect metrics from NGINX instances, including CPU, memory, and HTTP status codes. The blog provides a step-by-step guide on setting up a Grafana dashboard using the NGINX Instance Manager's Prometheus-compliant data store to display these metrics, offering a free pre-built dashboard to users. It suggests further customization of the dashboard to align with specific operational needs and hints at setting up alerts for critical metrics, encouraging the use of Grafana's REST API for automating deployments.

During Day 4 of GrafanaCONline 2021, several presentations showcased the versatile applications of Grafana in various fields, particularly focusing on real-time streaming in Grafana 8 for IoT projects. Ryan McKinley highlighted Grafana's enhancements for industrial and IoT use, including Grafana Live for broadcasting real-time data. Ed Welch and Ivana Huckova shared easy DIY IoT projects using Grafana, while Martin Klimmek and Haluk Tutuk discussed Siemens Mobility's use of Grafana for train data analysis. Grant Pinkos outlined how Grafana Cloud aids in improving manufacturing efficiency at American Metal Processing. Mat Schaffer from Safecast demonstrated how Grafana dashboards are used to visualize environmental data. Additionally, a panel discussed careers in open source, emphasizing the value of mentorship and community contributions in Grafana Labs.

Ivana Huckova and Ed Welch presented a session at GrafanaCONline showcasing easy DIY IoT projects that utilize Grafana, Arduino, Prometheus, and Loki, demonstrating that getting started with IoT can be both accessible and enjoyable. They provided resources, including a GitHub repository with instructions on configuring Arduino and Grafana Cloud, along with sending metrics and logs through Prometheus and Loki. Attendees can access Arduino libraries for seamless integration of metrics and logs from embedded devices, and the projects used Grafana Cloud for effective observation and dashboard creation. The session emphasized the simplicity of using Grafana Cloud, which offers free metrics, logs, and traces, and encouraged participants to share their own IoT projects.

GrafanaCONline 2021 Day 3 featured in-depth sessions on Grafana Tempo, its open-source distributed tracing tool, with significant improvements in scale and efficiency, as well as new features like autologging and upcoming capabilities such as native search and long-term storage for exemplars. Pernod Ricard showcased their use of Grafana and Loki for monitoring their global e-commerce Magento sites, detailing their AWS-based infrastructure and future growth projections. Jodie Barnsley presented her master's project using Grafana and InfluxDB to predict air quality, while lightning talks offered advanced tips for optimizing Grafana usage, including deploying with Tanka and using dynamic variables. Dapper Labs highlighted their journey of scaling their observability platform with Grafana, Cortex, Loki, and Tempo to manage 3 million metrics per minute, emphasizing the importance of a consistent monitoring stack despite significant changes in business needs. The recap also previewed upcoming sessions focused on IoT applications and environmental data visualization, offering opportunities for community engagement via Slack.

Grafana 8 introduced a unified alerting system, enhancing the alerting experience by integrating Grafana panel alerts and Prometheus-style alerts into a consistent system across Grafana OSS, Grafana Cloud, and Grafana Enterprise. This update involves a major transformation of the alerting engine, providing a common API and consolidating alert management onto a single interface. The new system allows users to create, edit, and manage alerts more efficiently, supports multiple panel types, and improves the organization of notification channels and silences. The enhanced interface offers more functionalities, such as multi-dimensional alert rules and advanced filtering options, while maintaining compatibility with legacy alerts. Currently available in Feature Preview mode, the new alerting system will eventually become the default, automatically migrating all existing alerts and deprecating older plugins upon its full release.

GrafanaCONline 2021 Day 2 showcased significant developments in Grafana 8.0 and innovations in Prometheus and Cortex, demonstrating how these tools enhance data visualization and observability. Highlights included an in-depth look at Grafana 8.0's new features, such as updated navigation and a metrics picker for Prometheus, alongside new panels like Time series and Bar chart. The conference also presented advancements in Grafana Enterprise, including improvements in reporting and access control, and introduced a variety of plugins that expand Grafana’s capabilities beyond time series data. Notably, Robinhood shared their experience scaling from 100 million to 1 billion time series using Cortex and Grafana Enterprise Metrics, emphasizing the critical role these tools played during the surge in GameStop trades. Additionally, sessions covered the use of Prometheus and Grafana at Roblox to answer internal queries efficiently and create comprehensive dashboards, while future sessions promised insights into distributed tracing, e-commerce monitoring, and air quality predictions with Grafana.

Grafana Tempo, an open-source distributed tracing backend, has reached general availability with its 1.0 release, marking a significant milestone in its development. Originally announced at ObservabilityCON in October 2020, Tempo has greatly improved its capacity from processing 150k spans per second to over 2 million spans per second, showcasing its enhanced performance and scalability. This achievement has resonated with power users managing vast trace volumes, while plans are in place to make it more accessible to medium and small users by adding native search capabilities. The release also coincides with the introduction of Grafana Cloud Traces, allowing users to integrate traces with other observability tools and offering up to 50GB of free trace uploads per month. The success of Tempo is attributed to the collaborative efforts of its development team and the wider community, who continue to contribute to its growth and improvement.

GrafanaCONline 2021 kicked off with a range of sessions highlighting the latest developments and applications of Grafana, including the release of Grafana 8.0 and the introduction of Grafana Tempo 1.0, a novel approach to distributed tracing. The event, led by key figures such as Grafana Labs CEO Raj Dutt and other executives, underscored the company's commitment to open-source software while balancing enterprise needs. Topics covered included the use of Grafana for monitoring ultraviolet sensors on the International Space Station, insights into remote work habits using IoT devices, and advancements in machine learning for predictive analytics and alerts. The conference also featured discussions with industry leaders, such as Salesforce, on the importance of availability and adaptability during the pandemic. Attendees were encouraged to engage with the Grafana community through various sessions, which are available on demand, and to participate in discussions on Slack.

GrafanaCONline 2021 showcased numerous updates and new features for Grafana, focusing on enhancing data visualization, security, and alerting functionalities. Grafana 8.0 introduced new panels, including a State timeline panel for observing discrete state changes, and the Time series panel became the default visualization, offering improved customization and performance. The introduction of Library panels allowed for more efficient dashboard management by enabling reusable panel updates. Enhanced security measures were also introduced, such as fine-grained access control and data source query caching, which optimize performance and reduce costs. Unified alerting consolidated Prometheus and Grafana dashboard alerts, simplifying management and access. Additionally, Grafana Tempo 1.0 was launched, offering a novel approach to distributed tracing with a focus on efficient trace storage, while better logs navigation and machine learning capabilities were added to improve system understanding and prediction capabilities. These updates were complemented by new plugins, a more accessible plugin catalog, and the announcement of Grafana Cloud Traces, enhancing the overall observability stack.

Grafana Cloud has introduced a new integration with NGINX that enhances observability by combining metrics, logs, and traces with Grafana, using Loki for log processing. This integration allows users to create a custom JSON access log for visualizing key metrics such as traffic, error rates, and visitor demographics without replacing existing logs. For those without an easily reconfigurable NGINX deployment, a demo application is available to set up the necessary components, including a sample HTML5 application and GeoIP2 IP address geocoding, hosted on a VPS or cloud provider. The integration is accessible to Grafana Cloud users, with both free and paid plans available, encouraging users to explore monitoring options and engage with the Grafana Community for further information and support.

Ryan Geyer, a Senior Cloud Integrations Engineer at Grafana Labs, is based in Lompoc, California, near the Vandenberg Space Force Base and Santa Barbara wine country. His role involves building Grafana Cloud integrations and user-friendly dashboards to facilitate a seamless onboarding experience for new users. While he hasn't contributed to major open-source projects, he has developed tools and 3D printable models for the Pokémon GO community. In his free time, Ryan enjoys cycling, 3D printing, and working on cars, and he is notably excited about the future of electric vehicles, particularly the Ford F150 Lightning. Passionate about helping users experience "a-ha" moments, he looks forward to reconnecting with friends post-Covid and enjoys any food featured on the sandwich alignment chart.

Grafana Cloud's new Humio plugin enables seamless integration of Humio log data with other data sources in Grafana, enhancing visualization and dashboard creation for organizations. This integration caters to DevOps, ITOps, and security teams by allowing real-time data aggregation, search, and alerting, while offering customizable dashboards that can be shared across departments for efficient issue resolution. Grafana Cloud provides a free tier and a 14-day trial of its Pro plan, which includes extended storage and collaboration features. The plugin supports the use of Humio's query language and variables for creating templates, facilitating efficient dashboard development. Schibsted, a Norwegian media group, utilizes this integration to provide consistent, insightful dashboards across their organization, enabling users to quickly derive business insights by correlating Humio log data with other metrics in a unified interface.

GrafanaCONline 2021 is set to bring the Grafana community together during the weeks of June 7 and 14, featuring over 70 speakers and the launch of Grafana 8.0 with new features and announcements. The event will include introductory workshops on Grafana, Loki, and Prometheus, which are free and require pre-registration, and will be held via Zoom. The official conference schedule kicks off with an opening keynote on June 8, showcasing a demo of Grafana 8.0 by its creator and a Q&A session with Grafana Labs CEO and Salesforce team members. Sessions will be streamed on Tuesdays, Wednesdays, and Thursdays, and attendees can participate in live Q&As through the GrafanaCONline event page. Attendees are encouraged to join the Grafana Labs Community Slack workspace to connect with others, ensuring a friendly and engaging environment while adhering to the Grafana Labs Event Code of Conduct.

Grafana Cloud has introduced a new integration for monitoring RabbitMQ, a widely used open-source message broker, enabling users to track essential cluster metrics through the Grafana Agent. This integration builds on existing community dashboards for RabbitMQ by adding additional filtering options and ensuring compatibility across various infrastructures, not just Kubernetes. Key features include monitoring cluster health, memory consumption, and node-level metrics, alongside packaged alerts for common issues such as high memory usage or unroutable messages. The integration is compatible with RabbitMQ version 3.8 and above, requiring no additional infrastructure, and is available to both free and paid Grafana Cloud users. Users are encouraged to engage with the Grafana Community on Slack for further information and support.