May 2020 Summaries

During the final week of GrafanaCONline, Grafana Labs introduced significant updates to Grafana Cloud Graphite and Metrictank, as detailed by principal software engineer Dieter Plaetinck, who highlighted features like meta tags for large-scale environments and tools for diagnosing and optimizing query performance. Additionally, a case study was presented by ML6, showcasing how they teamed with Accolade Wines to utilize Grafana and machine learning in order to significantly reduce wine waste, achieving a 200% return on investment and saving the company an estimated £75,000. The conference also featured sessions on community and documentation with Grafana Labs senior technical writer Diana Payton and a closing discussion about the business dynamics of Grafana Labs led by co-founder and CEO Raj Dutt, emphasizing the importance of community engagement through platforms like Slack.

Grafana 7.0 features significant UX improvements driven by user feedback and interdisciplinary collaboration, with a focus on enhancing the panel editing experience, introducing a privacy-friendly meta analytics feature, and establishing a consistent brand design. The redesign of the panel edit interface involved extensive user research to address issues like the organization of options and intuitive use, leading to a new layout that facilitates easier panel creation and modification. Additionally, the introduction of the meta analytics Enterprise feature aids in dashboard management by providing insights into dashboard usage without compromising user privacy. The update also included visual and branding enhancements with a new icon set and improved color consistency to ensure a cohesive user experience. Furthermore, Grafana Labs launched a documented design system to streamline component usage across community developers and internal teams, aiming to synchronize design mockups, interfaces, and plugins. These changes reflect Grafana Labs' ongoing commitment to user-centered design and continuous improvement based on community feedback.

GrafanaCONline Day 9 featured insightful sessions, including a detailed explanation by Grafana Labs principal software engineer Björn Rabenstein on Prometheus rate queries, where he delved into the distinctions between range vectors and instant vectors, as well as range and instant queries, using Grafana's Query Inspector, a new feature in Grafana v7.0, to illustrate how data from Prometheus is graphed. Additionally, Stig Sorenson from Bloomberg LP shared how his organization successfully implemented a centralized telemetry solution, involving more than 6,000 engineers and built on open-source software, shedding light on critical decisions and lessons learned over a four-year period. Attendees were also encouraged to participate in upcoming sessions and engage with the Grafana community on Slack for ongoing updates and discussions.

Integrating load and performance testing into a continuous integration pipeline can help detect performance issues early, and this guide offers insights into using Grafana k6 with Azure Pipelines for this purpose. k6 is a free, open-source tool that allows developers to write load and performance tests in JavaScript, easily integrating into their daily workflows. Azure Pipelines, part of Microsoft's Azure DevOps, supports continuous integration and delivery by testing, building, and deploying code. The guide suggests starting small with isolated tests and progressively expanding them, emphasizing the importance of clear service level objectives (SLOs) to assess system reliability. The document provides a practical example of a load test script using k6, demonstrating how to configure virtual user stages and set performance thresholds. Additionally, it explains how to execute tests locally and through Azure Pipelines, as well as how to use cloud execution for geographically distributed or resource-intensive tests. The integration is straightforward and can help maintain system performance by identifying regressions before they impact users.

GrafanaCONline Day 8 featured the introduction of a new plugins platform with the release of Grafana v7.0, aimed at allowing developers to create custom plugins more efficiently. Dominik Prokop from Grafana Labs detailed the new packages, including @grafana/ui, @grafana/data, @grafana/runtime, and @grafana/toolkit, and demonstrated a new declarative options API. The event also highlighted a presentation by Matti Paksula, who demonstrated using Grafana to visualize Chrome browsing data in near real time, emphasizing its use for tracking site performance metrics. The day's sessions included talks on Prometheus rate queries and strategies for organizational adoption of telemetry solutions, with opportunities for attendees to engage with the Grafana community on Slack.

GrafanaCONline's final week offers a series of online sessions, including discussions on Grafana plugins by Dominik Prokop and using Chrome browsing data with Grafana by Matti Paksula. Future presentations will cover topics like Prometheus rate queries, Grafana Cloud Graphite, and Metrictank. Participants can watch previous sessions on demand, engage with the Grafana community, receive updates from the Grafana Labs team, and participate in live Q&A sessions on Slack by joining the #grafanaconline channel.

Grafana 7.0 introduces significant enhancements aimed at improving observability in distributed microservice architectures by integrating trace viewers and data sources from Jaeger and Zipkin. These additions address the challenges associated with understanding complex systems and tracing the origin of errors, offering users a familiar and efficient UI. The trace viewer in Grafana allows detailed examination of traces, enabling users to zoom into specific regions and access span details such as tags or logs. The integration also supports basic querying by trace ID and allows internal linking with Loki derived fields, facilitating the correlation of data from various sources. As part of its ongoing development, Grafana plans to expand support for more data sources and integrations, ultimately striving to deliver a fully integrated observability experience that simplifies problem-solving while maintaining flexibility in data source selection.

GrafanaCONline Day 7 featured discussions on the evolution and future developments of Loki, a logging tool integrated with Grafana, and the art of storytelling through dashboards. Loki tech lead Ed Welch and maintainer Cyril Tovena presented on the project's history, current advancements, including query parallelization and enhancements to the LogQL query language, and future plans. In another session, Julien Pivotto, an open-source consultant and Prometheus maintainer, highlighted the importance of creating dashboards that effectively communicate stories through visualizations, offering practical tips on using Grafana's advanced features for clear and accessible data representation. The event encourages community interaction and provides updates via Slack for participants.

Cortex v1.1 has been released, bringing enhanced reliability and performance through contributions from more than 30 authors across 9 companies. Key features of this release include support for the Prometheus /api/v1/metadata API, allowing Grafana Agent to send metric metadata to Cortex for use in Grafana's Explore view, and the promotion of the Write-Ahead-Log (WAL) feature to production-ready status after extensive testing. The release introduces a significant optimization for regular expression selectors, achieving up to a 100x improvement in query performance by eliminating the need for regular expressions during index lookups in the ingester. Improvements have also been made to the Cassandra chunk storage backend, including TLS host verification and limiting read concurrency, and the query frontend component has been embedded into the single-process Cortex deployment to simplify configuration. Additionally, Cortex has integrated a block-storage approach from Thanos, enhancing the scalability of the read path with the introduction of a store-gateway service, which facilitates horizontal scaling by sharding and replicating blocks across all store-gateway instances. The release reflects increasing interest and development activity in Cortex, supported by the community and leveraged in Grafana Cloud's Prometheus service.

Day 6 of GrafanaCONline featured insightful presentations highlighting the versatility of Grafana in different contexts. Malcolm Holmes from Grafana Labs showcased Tanka, illustrating how it simplifies managing Kubernetes resources with Jsonnet, allowing dashboards to be deployed with minimal code. This session underscored the advantages of integrating dashboard deployment with code management. Meanwhile, Matthias Mehldau from The Hiveeyes Project demonstrated how Grafana dashboards can aid beekeepers by monitoring hives and weather conditions, and by facilitating comparisons of particulate matter sensor data across networks, as well as correcting readings from low-cost humidity sensors. These sessions exemplified Grafana's potential for innovative applications, and attendees were encouraged to stay engaged with the community and upcoming sessions through Slack.

GrafanaCONline Day 5 featured two key presentations, one by Ed Welch and another by Jonan Scheffler, highlighting innovative uses of Grafana and Kubernetes. Ed Welch, a Senior Software Engineer at Grafana Labs, showcased how he maximizes the range of his electric Nissan Leaf by analyzing driving behavior through data collected via Grafana, Cortex, and Loki on a Raspberry Pi. He processes over 1,000 messages per second from the vehicle CAN bus, GPS data, and camera images to receive real-time feedback and perform further analysis, with plans to adapt this method for non-electric vehicles. Jonan Scheffler, a developer advocate, demonstrated the process of creating a desktop Kubernetes cluster using Raspberry Pis, explaining the roles of containers, pods, and nodes, and providing practical advice for using k3sup to bootstrap Kubernetes. The event also included other sessions on topics like Tanka and particulate-matter readings, with opportunities for attendees to engage with the Grafana community via Slack.

The release of Loki version 1.5.0 introduces a significant change with the boltdb-shipper index, allowing the software to operate without a separate dedicated index store, thus simplifying deployment and reducing operational costs. This new feature, developed by Sandeep Sukhani and his team, uses an in-memory boltdb index to send snapshots to an object store like S3 or GCS, making the sharing of index information more efficient. However, the upgrade also includes substantial changes due to an updated Cortex library dependency, which may lead to config errors for existing setups, prompting operators to review and adjust their configurations accordingly. Additionally, the Loki Docker image now runs under a different user, which might require modifications to file permissions and directory mounts. The release includes other enhancements and bug fixes, with nearly 140 pull requests merged, and further details and future features are discussed in an upcoming GrafanaCONline talk.

GrafanaCONline Days 3 and 4 focused on the release of Grafana 7.0, highlighting new features such as UX enhancements, a unified data model, and a new plugins platform. The event showcased how Grafana has transformed the oil and gas industry by enabling efficient monitoring and management of remote assets through dashboards and alerting, which has led to improved decision-making at Whiting Petroleum. Prometheus' future was discussed by Goutham Veeramachaneni, who covered recent developments and upcoming features, including server-side metric metadata support and Open Metrics adoption. David Worth from Strava delved into the essential observability concepts and tools for modern service architectures. The conference also featured presentations on diverse topics like Nissan Leaf's cloud integration and monitoring Kubernetes with Raspberry Pis, fostering community engagement via Slack and offering on-demand session access.

Grafana 7.0 introduces a host of significant enhancements aimed at improving user experience, plugin development, and data transformation capabilities. The release features a new plugin architecture that simplifies the creation and management of plugins using a React component library and a CLI toolkit, making it accessible for both seasoned users and newcomers. Visualizations have been enhanced with a unified data model, allowing for more consistent data configurations and a revamped Panel Editor that facilitates easy customization. Tracing support is integrated to help users navigate complex microservices environments by connecting and visualizing metrics, logs, and traces seamlessly. The update also includes a new transformations capability that allows users to manipulate non-time-series data and combine it with existing datasets, alongside an enhanced Panel Inspector for data inspection and export. Additionally, AWS CloudWatch Logs support and enterprise-level usage analytics provide deeper insights into dashboard utilization and user behavior, making Grafana 7.0 a robust tool for monitoring and data analysis.

GrafanaCONline Day 2 featured insights into the future of worldPing and an industrial IoT use case of Grafana, with presentations highlighting both technical advancements and practical applications. Grafana Labs co-founder and CTO Anthony Woods discussed enhancements in worldPing, an open-source plugin for monitoring global Internet application performance, and previewed features like metric and log collection in the upcoming worldPing v2. Additionally, René de Theije from Datask demonstrated how Grafana is used in industrial IoT to monitor fillet machine production in the Netherlands, showcasing the tool's scalability and effectiveness in real-time industrial automation. These sessions exemplified Grafana's versatility and its growing role in both global performance monitoring and industrial applications.

GrafanaCONline's first day featured a keynote from Grafana Labs leaders, including creator Torkel Ödegaard, CEO Raj Dutt, and VP of Product Tom Wilkie, who shared the history and future of Grafana Labs and its community. Highlights included the evolution of Grafana since its inception on January 1, 2014, and the anticipation of the 7.0 release, alongside updates on the Grafana Cloud metrics and logging SaaS platform components like Loki, Cortex, and Grafana Agent. Notably, active Grafana installations have surpassed 550,000. The keynote and other sessions are available on-demand, allowing participants to engage with the Grafana community and access a full schedule of sessions, including presentations on worldPing and Industry 4.0 implementations.

GrafanaCONline, a virtual conference, begins today, featuring a keynote by Grafana creator Torkel Ödegaard, Grafana Labs CEO Raj Dutt, and VP of Product Tom Wilkie, who will discuss the project's evolution and future. The event celebrates significant milestones, including over 550,000 active Grafana installations, 60 trillion samples processed monthly on Grafana Cloud Prometheus, and 500 TB of data managed monthly on Grafana Cloud Loki. Attendees can look forward to teasers about the upcoming Grafana 7.0 release and have the opportunity to engage with the Grafana community on Slack. The conference offers a series of sessions over the next weeks, with on-demand recordings available for those unable to attend live, ensuring access to all presentations after a one-time email sign-in.

Loki, an open-source logging system inspired by Prometheus, addresses challenges faced by traditional log management tools by offering a cost-effective and scalable solution. Unlike systems like Elasticsearch that require extensive indexing, Loki only indexes a small amount of metadata from each logline, making it cheaper and more efficient for high-volume log aggregation. This approach aligns well with Prometheus metrics and is particularly effective for debugging and troubleshooting within Kubernetes environments. Since its inception in 2018 by Tom Wilkie of Grafana Labs, Loki has gained significant traction in the open-source community due to its integration with Grafana and its suitability for handling petabyte-scale log data. Users can quickly get started with Grafana Loki, and it offers both free and paid plans through Grafana Cloud, catering to various use cases and enhancing log analysis capabilities.

Grafana 7.0 introduces a significant enhancement with the new panel inspector feature, allowing users to view raw data in table format, apply transformations, and download it as a CSV file, expanding the capability previously limited to the Graph & Table panel. This feature is supported across all panels, including both internal and external community plugins, thanks to the unified data model and processing in version 7.0. Additionally, Grafana 7.0 offers other updates such as auto grid layout, a new table panel, and query history in Explore. The beta version of 7.0 is currently available for download, and further announcements are expected during the upcoming GrafanaCONline event, offering opportunities for users to engage with the latest developments.

Stephanie Closson is a software developer at Grafana Labs, working on the plugins and integrations team, where she expresses immense enthusiasm for the product and contributes to various open-source projects such as the OPC UA plugin and the Grafana DIV panel. She is particularly interested in visualizing geospatial data and making Grafana accessible to industries reliant on OPC technologies. In her free time, Stephanie enjoys mountain climbing and trail running with her son, aiming for 25 summits this year after achieving 17 last year. She also highlights her unique accomplishment of completing the "Canadian Death Race" with her spouse, and has a background in music, having fulfilled the requirements for a music minor and playing the alto saxophone.

Grafana's upcoming 7.0 release will replace the PhantomJS image renderer with the new Grafana Image Renderer plugin, addressing the numerous issues associated with PhantomJS, including bugs and lack of support for modern web technologies. The change is driven by the end of PhantomJS project maintenance and aims to enhance the reliability and security of rendering images of Grafana dashboards. The new plugin, which has been available since June 2018, operates using a headless browser like Chromium and can be installed as a Grafana plugin or run as a remote rendering service. Docker is recommended for running the remote service, and several configuration options allow for scaling and optimizing performance, including setting up clustered modes to manage concurrent requests. Despite some challenges, such as lack of support for ARM architectures and the necessity of IPv6, the Grafana team continues to improve documentation and support, while Grafana Cloud users benefit from hosted rendering services. The release is part of a broader update with new features and best practices, promising better integration and performance as Grafana prepares for its 7.0 launch.

Grafana Labs was recognized as one of Inc.'s 2020 Best Workplaces, a distinction based on an employee survey by Quantum Workplace that evaluated trust, management effectiveness, perks, and confidence in the company's future. Among over 3,000 submissions, Grafana Labs stood out, with a notable 98% of employees, known as Grafanistas, expressing job satisfaction and pride in their workplace. The company attributes its success to being a remote-first organization, a model they embraced well before the COVID-19 pandemic, promoting productivity and inclusivity, and has shared insights to aid others in transitioning to remote work. The recognition from Inc. is celebrated, but the overwhelming employee satisfaction is highlighted as the more significant achievement.

Grafana Labs was recognized as one of Inc.'s 2020 Best Workplaces, a distinction determined from over 3,000 submissions through an employee survey by Quantum Workplace assessing trust, management effectiveness, perks, and confidence in the company's future. With 98% of Grafana Labs employees expressing job satisfaction and pride, the recognition highlights the company's successful remote-first work model, which was in place even before the COVID-19 pandemic made remote work widespread. Grafana Labs is enthusiastic about this acknowledgment and continues to share insights on effective remote working practices to support other organizations during their transition to remote work environments.

Prometheus 2.17 introduced isolation to its time series database (TSDB), addressing a significant limitation present in previous versions where concurrent queries and scrapes could lead to inconsistent results. Prior to this update, queries might only see a portion of the samples, particularly affecting functions like histogram_quantile, which rely on consistent data from histograms. The feature was initially conceptualized and partially implemented by Prometheus team member Brian Brazil but was delayed due to overlooked pull requests. The isolation feature, which ensures that queries view a consistent state of data, was eventually completed with contributions from various developers and finally integrated into Prometheus 2.17. While the introduction of isolation slightly increased CPU usage, memory usage, and query latency, these changes are generally negligible in most scenarios. The update ultimately resolved the issues where rule evaluations could yield inaccurate results due to uncoordinated data states, marking a significant improvement in the reliability of Prometheus queries.

Robert Fratto, a software engineer at Grafana Labs based in Orlando, contributes significantly to various open-source projects such as Loki, Cortex, the Grafana Agent, and Prometheus, with recent work focusing on enhancing the scalability of the Grafana Agent using the Cortex ring. He joined Grafana Labs because of his admiration for their software and a desire to work with developer-oriented products. In his free time, Robert engages in side projects, video gaming, and reading, with a particular interest in creating a Twitch.TV clone, playing Final Fantasy VII: Remake, and exploring the Wheel of Time series. He appreciates the intelligent colleagues he works with and enjoys learning from them daily. His favorite foods include pizza and, more candidly, Oreos.

Grafana 7.0, set to be released next month, introduces a query history feature in Explore, allowing users to view, manage, and reuse past queries efficiently. Each query in the history can be executed, commented on, copied to the clipboard, or starred for quick access, helping users reuse their favorite queries without retyping them. Additional previews of Grafana 7.0 include features like auto grid layout, a new table panel, and a panel inspector. A beta version is currently available for download, and more announcements will be made during the upcoming GrafanaCONline event, for which users can sign up to receive updates and tune in to live presentations.