May 2019 Summaries

At KubeCon + CloudNativeCon EU in Barcelona, Grafana Labs made a notable impact by delivering four talks and a keynote to an enthusiastic audience. The keynote, presented by Grafana Labs VP of Product Tom Wilkie and Red Hat’s Frederic Branczyk, focused on the future of observability, highlighting new advances in the field. One of the featured technologies was Grafana Loki, presented by Tom Wilkie, which is designed to handle logs similarly to how Prometheus handles metrics, and includes a new query language called LogQL. Additionally, Tom Wilkie, along with Bryan Boreham from Weaveworks, introduced Cortex, explaining its ease of setup and its operation as a microservice, while also offering a deeper dive into its cloud storage options and performance tuning. David Kaltschmidt, Grafana Labs Director of UX, showcased effective dashboarding techniques for Kubernetes, aimed at aiding sleep-deprived on-call personnel.

Grafana Labs has transitioned from contributing to Cortex, a scalable storage solution for Prometheus, to launching Grafana Mimir, which allows for greater scalability. At KubeCon + CloudNativeCon in Barcelona, a session by Grafana Labs' Tom Wilkie and Weaveworks' Bryan Boreham showcased optimizations to Cortex's query performance, highlighting how the integration of the Prometheus PromQL engine with a scale-out storage engine ensures feature compatibility with Prometheus queries. The optimizations include batch iterators for merging results, extensive caching to reduce query latency, query parallelization, and the introduction of a query frontend that aligns, splits, caches, and queues queries. These enhancements have led to significant improvements, with Cortex achieving average response times of less than 50 milliseconds in Grafana Cloud. Despite the progress, further collaborations, particularly with the Thanos team, are planned to enhance performance for high-cardinality workloads, reflecting the project's evolution from focusing on ingestion challenges to prioritizing query performance.

At KubeCon 2019, Grafana Labs' Director of UX, David Kaltschmidt, emphasized the importance of creating foolproof Kubernetes dashboards to alleviate the stress of on-call shifts. Drawing from personal experiences, Kaltschmidt highlighted the challenges of managing on-call duties, particularly when dealing with complex systems like Kubernetes. To address these issues, he introduced the Dashboarding Maturity Model (DMM), which provides a structured approach to improving dashboard consistency and effectiveness within organizations. The model outlines three tiers of maturity, with strategies to address common pitfalls such as dashboard sprawl, lack of version control, and inefficient navigation. For higher maturity, Kaltschmidt recommended using templated variables, hierarchical dashboards, and scripting libraries to ensure consistency and ease of use. He also stressed the importance of expressive dashboards that use color and normalization to quickly convey system states, aiming to reduce cognitive load and improve troubleshooting efficiency. Future developments for Grafana include enhancing the provisioning workflow and introducing an in-browser JSON editor to streamline dashboard management.

At KubeCon + CloudNativeCon EU, Grafana Labs VP of Product Tom Wilkie and Red Hat Software Engineer Frederic Branczyk discussed the evolving landscape of observability beyond the traditional three pillars of monitoring, logging, and tracing. They proposed the integration of these pillars for more seamless workflows, highlighting tools like Grafana Loki and OpenTelemetry for enhanced data correlation. They also predicted the emergence of new signals and analyses, such as time series profiling, which could provide deeper insights into system performance issues. Additionally, they forecast a shift towards simpler, index-free log aggregation systems, exemplified by tools like Grafana Loki, which prioritize ease of use and scalability for developers. The talk emphasized community involvement in advancing observability practices to create more reliable systems.

Grafana Labs' presentation at KubeCon EU by VP Product Tom Wilkie focused on Loki, a log aggregation system designed to work seamlessly with Grafana and inspired by Prometheus. Loki aims to optimize log search, aggregation, and exploration by adopting a data model that uses tags instead of traditional inverted indexes, making it both scalable and cost-effective. This model results in a significantly smaller index size, allowing for efficient operations even in large environments. Loki integrates with existing observability stacks, simplifying the process of correlating metrics and logs, and is designed to be developer-friendly by enabling offline development on a single binary. Grafana Labs also offers a hosted version of Loki using a microservices architecture for cloud-native environments. Future developments include the introduction of LogQL, a query language that will enable the combination of Prometheus and log queries, enhancing Loki's alerting capabilities.

Grafana version 6.2 introduces a range of new features and improvements, focusing on security, user interface enhancements, and performance optimizations. Key updates include the encryption of datasource passwords by default to enhance security, the introduction of a new Bar Gauge Panel with unique display modes, and support for Elasticsearch 7. The release also features lazy loading of out-of-view panels to reduce backend data source load, improved table data support in new react panels, and provisions for enhanced configuration handling through environment variables. Additionally, the update addresses minor bugs and introduces several minor features and fixes across various integrations like Azure Monitor and InfluxDB, while removing support for an old deprecated package repository. Users are encouraged to review upgrade notes for a smooth transition and can explore new features through demo dashboards.

Grafana Labs has transitioned its focus from Cortex to Grafana Mimir, an open-source long-term storage solution for Prometheus, although they continue to support and develop Cortex, a Prometheus-compatible time series database. Cortex is designed for high scalability and availability, enabling centralized aggregation of metrics from multiple Prometheus instances across geographically distributed Kubernetes clusters. It supports durable long-term storage using various cloud services, aiding in capacity planning and trend analysis, and offers multitenancy to reduce operational overhead. Recent enhancements include a single-process architecture for ease of use, a parallelizing query engine that significantly improves query latency, and support for highly available Prometheus pairs to enhance redundancy. Despite challenges in continuous deployment due to its stateful nature, Grafana Labs is committed to regular updates and backward compatibility, with plans to release a 1.0 version with features like a write-ahead log for increased durability.

Verizon, a major player in communications and media, successfully leveraged Grafana to enhance its monitoring capabilities, streamline operations, and facilitate self-service for end users. Faced with inefficiencies and high costs from its existing infrastructure, Verizon's Systems Engineering team transitioned from a sprawling, outdated system dependent on static SQL servers to a more flexible, dynamic setup using Grafana and MySQL. This shift reduced the lines of code needed for operations from 500,000 to just 500, significantly improving scalability and efficiency. The transition to Grafana empowered users across the company, from call center representatives to executives, by providing them with real-time data access and customizable dashboards, reducing the time required for report generation and incident response. Additionally, the new infrastructure decreased late-night alerts and helped engineers focus on more strategic tasks, enhancing overall productivity. This move not only improved internal processes but also set a foundation for future innovations, allowing Verizon to potentially contribute to Grafana's open-source community.

Johannes Schill, a frontend developer at Grafana Labs, is based near the Stockholm office and contributes to the Grafana open-source project. Despite his preference for tabs over spaces, his coding environment varies between silence, music, or ambient noise depending on the task at hand, and he finds focus through coffee, music, and humor, such as an animated gif of Nicolas Cage. In his personal life, Schill enjoys music, collecting vinyl records, and is a long-time attendee of Djurgårdens football games. Although his current free time is limited due to family commitments, he engages in running and snow sports when possible, with skiing and snowboarding being his preferred winter activities. Schill is not active on Twitter but can be found on GitHub under the handle "jschill."

Public Grafana dashboards offer a glimpse into how various organizations utilize Grafana for their data visualization needs, showcasing live data from large entities. GitLab, known for its transparency, shares dashboards of their cloud infrastructure, ranging from disk stats to alert reporting. Wikimedia's dashboards reveal the massive scale at which Wikipedia operates, displaying data from global datacenter overviews to API request rates. The Cloud Native Computing Foundation utilizes Grafana for its DevStats tool to analyze GitHub activity, tracking contributions and engagement levels. Grid Computing Centre Karlsruhe and CERN use Grafana to visualize data from their scientific experiments, including those related to the Large Hadron Collider. Grafana plugins for systems like Zabbix and IoT frameworks, as well as projects like Hiveeyes and Percona, demonstrate diverse applications and monitoring capabilities. The Grafana Play dashboard serves as a demo site to introduce Grafana's features and capabilities while also being used for testing and demonstrations by Grafana Labs.

Sensu offers a cost-effective and customizable monitoring solution that integrates with InfluxDB and Grafana to streamline infrastructure monitoring. Sensu's monitoring event pipeline allows users to collect, process, and route various event types such as discovery, availability, telemetry, and alerts, with its embedded StatsD daemon handling metric aggregation. Sensu service checks, which operate similarly to Nagios checks, generate metrics that can be visualized through Grafana, leveraging Sensu's support for multiple metric formats and extensive plugin capabilities. The integration with time series databases like InfluxDB, facilitated by a Sensu InfluxDB handler, allows for enriched metric tagging and seamless storage of event data. Sensu's dynamic and reusable pipeline supports multiple event types and offers a centralized entry point for diverse telemetry data, enhancing visibility across applications, systems, and infrastructure. The Sensu dashboard, while backend- and API-driven, works in conjunction with Grafana to provide a sleek visualization of critical metrics, demonstrating the power of open-source tools in monitoring complex infrastructures.

Trapeze Group's Emergency Services team has implemented Grafana to enhance the monitoring capabilities of ambulance operations in Australia, which involves managing a fleet of up to 1,000 vehicles with over 60 telemetry channels and 120 million daily messages. The integration of Grafana facilitates real-time visibility and accessibility of critical data, allowing for efficient tracking of ambulance locations, vehicle performance, and response times, ultimately improving life-saving operations. Initially facing challenges with bespoke development approaches, the team found Grafana's dashboards and plugins to be game-changing, enabling rapid prototyping and real-time data synchronization without heavy reliance on developers. The team continues to explore enhancements such as live data sources, predictive modeling for vehicle maintenance, and advanced mapping features, while expressing interest in technologies like Apache Spark, Prometheus, and InfluxDB to further improve data ingestion and analysis. Despite initial resistance from the development team, Grafana has proven to be an effective tool for visualizing and managing complex data, supporting Trapeze's goal of providing emergency services with comprehensive and intuitive monitoring solutions.

Andrej Ocenas is a fullstack developer at Grafana Labs, currently focusing on backend development while also engaging in React and frontend tasks. Based in Bratislava, Slovakia, he contributes primarily to the Grafana open-source project and has previously worked on various other open-source endeavors, including the React animation library and iOS Swift/Objective-C projects. Outside of work, Andrej enjoys snowboarding and researching snowboarding gear, exercising, and figuring out time-efficient healthy eating strategies. He is an avid reader of sci-fi books, often using Audible to listen to them while multitasking, and recently binge-watched "Love, Death & Robots" on Netflix, which he highly recommends. He humorously identifies with Bruce Banner from "Avengers: Infinity War."

In April, Grafana released version 6.1.5, which addressed a significant security vulnerability that was overlooked due to a process error where a crucial patch from version 5.3.3 was not merged back to the master branch, leaving subsequent versions exposed until 6.1.4. This oversight was identified and corrected with the new patch released on April 29, prompting Grafana to conduct an incident review to share the timeline, factors contributing to the error, and future security release plans. The company encourages feedback and provides a dedicated email for reporting security vulnerabilities, emphasizing the importance of encrypting such communications. Additionally, Grafana maintains a Security Announcements category on its community site to disseminate information about patches, their remediation, and mitigation details, offering users options to subscribe to updates via email or RSS feed.

Brendan Burns, co-creator of Kubernetes, shared insights at GrafanaCon L.A. into the evolution of monitoring cloud-native systems, highlighting the importance of decoupling through containers to simplify complex stacks and enable specialized focus on tasks like monitoring. Containers allow developers to expose metrics for monitoring without needing deep interaction, thus empowering both application developers and those focused on monitoring. Burns emphasized the necessity of standardization in monitoring tools, as seen with Kubernetes, to achieve consistency and expertise across teams. He cautioned against the pitfalls of snowflake clusters and advocated for policy enforcement to ensure uniform monitoring experiences. Through anecdotes, Burns underscored the importance of investigating anomalies in data, maintaining release dashboards to prevent outdated software issues, and employing blackbox monitoring to detect blind spots. He warned against the allure of flashy demos that may not reflect real-world utility and stressed the future need for default monitoring in packaged cloud applications, advocating for building reusable components to broaden industry participation and efficiency.

Loki, developed by Grafana Labs, is a log aggregation tool designed to optimize the search, aggregation, and exploration of logs within the Grafana interface, offering a cost-effective and streamlined alternative to traditional log management systems. Unlike other systems that require extensive data storage, Loki indexes only the data obtained from service discovery, allowing for a significantly reduced storage footprint. It operates as a multi-tenant, horizontally scalable microservice, enabling seamless correlation between logs and other observability data, akin to Prometheus, by attaching metadata labels through service discovery. Loki's primary focus is on incident investigation rather than replacing comprehensive analytics tools like Splunk or ELK. It provides a unified interface for correlating metrics and logs, simplifying the troubleshooting process by facilitating ad-hoc queries in Grafana Explore, thus aiding in identifying and resolving issues efficiently.

Grafana Labs addresses a common inquiry about whether to run Prometheus in a containerized environment versus a VM-based one, suggesting that if you are monitoring services in Kubernetes, utilizing Prometheus as a container within Kubernetes is preferable due to the need for direct connectivity for metric scraping. Prometheus benefits from being close to monitored services, and in scenarios with multiple private networks, multiple Prometheus servers may be necessary. While Prometheus itself is versatile and can run effectively in either environment, certain components like node_exporter may require running directly on the host due to their need for kernel interfaces. To ensure reliability even if the Kubernetes cluster encounters issues, implementing a "Dead Man's Switch" alert can notify an external service to confirm that Prometheus and the alerting pipeline are functioning correctly.

Leonard Gram is a backend developer at Grafana Labs, currently leading efforts to enhance Grafana Enterprise while primarily contributing to Grafana's open-source projects. Based in Stockholm, Sweden, Leonard is also known for creating GroovyCSV, although it requires minimal maintenance now. In his personal time, he enjoys playing board games, trading card games, and spending time with his adventurous small dog. Leonard has recently taken to using i3, a tiling window manager, to better manage his applications, and he stays focused while coding by minimizing distractions with techniques like jotting down unrelated thoughts in a notebook and creating a sense of urgency. His GitHub and Twitter handles are both xlson.

Grafana's tutorial on creating kiosks for displaying dashboards on a TV outlines the steps to set up Grafana dashboards using a Raspberry Pi or similar devices in kiosk or TV mode, providing a visual and automated display solution. The tutorial addresses the lack of comprehensive documentation on this feature and guides users through the process of logging into Grafana servers, switching display modes, and starting playlists. It includes instructions on configuring a Raspberry Pi, installing necessary utilities, and setting up secure remote access via SSH. The tutorial also covers cloning the Grafana kiosk Git repository and running specific commands to display Grafana dashboards on a TV, whether using a local, hosted, or anonymous Grafana login. The guide aims to simplify the process of creating engaging and informative visual displays, encouraging users to showcase their setups on social media.

AMMP Technologies is leveraging the Internet of Things (IoT) and Grafana to facilitate electrification in rural Africa by implementing solar mini-grids, as demonstrated in a Tanzanian village that gained electricity access in 2017, significantly enhancing local living standards and economic activities. These solar mini-grids comprise solar panels, power electronics, batteries, and smart meters, enabling efficient power distribution and consumption monitoring. AMMP monitors over 100 such grids across Africa, using data analytics to optimize grid operations and maintenance without the need for frequent in-person interventions, thus overcoming challenges posed by remote locations. The data collected is stored in an Influx database, analyzed with tools like Kapacitor, and visualized using Grafana, which helps in identifying and resolving technical issues remotely, such as voltage drops or cable damages. Additionally, AMMP capitalized on excess solar energy to support a water pumping project in Changombe, further demonstrating how data-driven solutions can enhance resource utilization and improve community infrastructure.