August 2020 Summaries

In an effort to help developers enhance their Grafana experience through custom visualizations and data source integrations, a live webinar titled "All about plugins: Building your first panel plugin" is being hosted by Developer Advocate Marcus Olsson and Solutions Engineer Christine Wang on September 2. The session will focus on setting up Grafana for plugin development, utilizing Grafana Toolkit to expedite the process, understanding different plugin types, and creating custom visualizations. Aimed at participants with basic web development experience, the webinar promises to provide valuable insights into building high-quality custom plugins, leveraging the new plugins platform introduced in Grafana v7.0. Attendees are encouraged to register for the event and explore popular community plugins for inspiration beforehand.

Heds Simons is a Solutions Engineer at Grafana Labs, where he leverages his expertise with Grafana, Loki, and Prometheus to assist potential customers in solving their infrastructure challenges. After emigrating from the UK to Canada, he now resides on Vancouver Island and works on Pacific Standard Time. With over two decades of engineering experience, Heds enjoys the diverse responsibilities of his role, which range from writing blog posts to running product trials and collaborating with various teams to enhance customer experiences. He contributes to open-source projects, notably in the balena OSS IoT ecosystem, and has personal projects involving IoT systems for monitoring beehives. In his leisure time, Heds indulges in hobbies such as gaming, hiking, canoeing, and cooking, and he recently upgraded his ice cream-making skills with a new Lello Musso Lussino ice cream maker. He finds joy in helping customers optimize their systems and appreciates engaging with them to improve their technological solutions.

Loki, a logging system developed by Grafana Labs, emphasizes the use of minimal labels to achieve better performance and lower operating costs, differing from traditional logging systems that rely heavily on indexing. This approach leverages a small index to enhance efficiency and employs horizontal scaling and query time brute force to manage large volumes of data, making it easier to run than fully indexed solutions. While indexing fewer labels might seem counterintuitive, it helps keep the system cost-effective and less complex. Although Loki cannot aggregate on non-label content yet, upcoming updates to its query language, LogQL, aim to address this limitation. The system can manage vast data volumes effectively by parallelizing searches, allowing users to maintain a small index even while handling terabytes of logs. However, challenges remain in environments like Lambda functions, where log ordering can pose issues, and efforts are ongoing to improve this aspect.

Grafana Labs has enhanced its platform with the release of Grafana v7.0, which has encouraged developers to create custom plugins that improve the visualization and integration of data sources, elevating the Grafana experience. These plugins, developed by the community, allow users to share tools that offer unique visualizations and data source integrations. Notable panel plugins include Boom Table, which features multi-level thresholds and customizable cell values; FlowCharting, which uses draw.io for complex diagrams; Diagram, which leverages mermaid.js for flow-charts; Status Panel for monitoring server stats; D3 Gauge, noted for its customizable gauges; and Discrete, which displays system states over time. Additionally, data source plugins like Redis and Google BigQuery enhance Grafana's flexibility by providing customizable monitoring solutions and support for sophisticated database features, respectively. Through these community-driven innovations, Grafana continues to enable users to present data in a way that aligns with their needs and preferences.

Björn "Beorn" Rabenstein's talk at KubeCon + CloudNativeCon EU 2020 focused on the future of Prometheus histograms, proposing a solution to existing issues by introducing histograms with effectively infinite buckets arranged in a logarithmic schema. This approach, which handles sparseness efficiently, has been used in other systems but presents challenges within the Prometheus model. Rabenstein has conducted research using Grafana Labs Cortex clusters and developed proof-of-concept implementations for instrumentation and encoding in Prometheus. While confident in overcoming initial challenges, further work is needed to optimize querying and assess the cost of storing and querying these histograms. The adoption of high-resolution, sparse histograms will require significant changes across the Prometheus stack, but Rabenstein is optimistic about its potential benefits and seeks community input for further development.

Jess Müller's guidance on improving Grafana dashboard design emphasizes the importance of simplicity and accessibility to enhance user experience. She suggests limiting the number of visualization types to facilitate quicker data comprehension, as diverse graph types can complicate interpretation and increase cognitive load. Müller also highlights the necessity of color accessibility, noting that color-blind users can be excluded if dashboards rely solely on color indicators without additional qualitative cues. She recommends using tools like accessibility checker plugins to ensure color choices are inclusive. Additionally, she advises using a consistent and limited color palette to maintain readability and direct attention effectively, utilizing salient colors for critical information. Müller encourages using company brand colors to create a more cohesive and professional appearance and invites readers to learn more by joining her webinar on dashboard design fundamentals.

Dasha Nikolov, a Talent Manager at Grafana Labs, recently relocated to Zurich, Switzerland, and operates within the Central European time zone. Embracing a social media-free lifestyle for her well-being, Dasha enjoys collaborating with engineering teams at Grafana Labs, a remote-first and globally distributed company, appreciating the diversity of interactions with people worldwide. Her role is driven by the inspiring passion of her colleagues and the opportunity to engage with varied cultures and stories, which she finds invaluable. In her free time, Dasha enjoys recreating restaurant recipes, exploring mixology, and DJing, a passion rooted in her love for '90s electronic music. A notable personal achievement for her was moving internationally with a large dog during the pandemic, a journey she finds rewarding and is willing to advise others on.

Cortex, a scalable Prometheus project, has advanced to the incubation stage within the Cloud Native Computing Foundation (CNCF), indicating its growing maturity and adoption among enterprises. Initially developed by Grafana Labs and now supported by a broad community, Cortex offers a centralized architecture and native multi-tenancy, making it well-suited for large-scale Prometheus implementations. The project's recent 1.0 release signifies its stability and ease of use, with contributions from over 79 companies and collaborations with projects like Thanos to enhance functionality. Grafana Labs launched Grafana Mimir to provide long-term storage solutions, further supporting Cortex's scalability to over a billion metrics. Cortex powers Grafana Cloud Metrics, offering scalable and fast Prometheus-compatible metrics and alerting services. The continued partnership with Thanos has led to shared advancements, reducing dependencies and improving performance, while new features like downsampling and support for exemplars are in development, promising a bright future for Cortex.

At KubeCon + CloudNativeCon EU, Grafana Labs engineer Marco Pracucci and Hashicorp's Thor Hansen discussed advancements in scaling Prometheus with Cortex, focusing on integrating Thanos technology to enhance Cortex's storage capabilities. The presentation outlined a new storage engine within Cortex, which reduces operational costs while maintaining performance and scalability. They introduced the experimental Cortex blocks storage, which eliminates the need for a separate index store by utilizing Prometheus TSDB per tenant, thereby storing all data in an object store. This approach leverages Thanos' principles, incorporating components like a compactor and store-gateway to manage and optimize data storage and retrieval. The talk highlighted Cortex's evolution, including its microservices architecture, single binary mode, and improvements in query performance through parallelization and caching strategies. The session emphasized the experimental nature of the blocks storage, its ongoing development, and the expectation of stabilization in the near future. The team also discussed future plans, such as optimizing query performance, implementing shuffle sharding for scalability, and collaborating with the Thanos community on data deletion features.

A live "Ask Me Anything" webinar featuring Torkel Ödegaard, the creator of Grafana, is scheduled for tomorrow, offering participants the opportunity to get answers to a wide range of questions about Grafana, from technical issues to user experience and new plugins. The session encourages attendees to inquire about best practices for managing dashboards when dealing with large amounts of data from various devices. Interested individuals must register and submit their questions before the deadline at 8 a.m. CET/6:00 UTC on August 19.

Grafana Labs recently announced a $50 million Series B funding round, building on a prior $24 million investment, to accelerate research and development and expand its engineering team, with a focus on product innovation and community growth. As part of this initiative, the company launched the Grafana Accelerator Program (GAP), designed to support innovation within the Grafana ecosystem by offering resources such as Grafana Cloud accounts, Grafana Enterprise subscriptions, cash grants, equity financing, and access to core developers for early-stage companies and projects using Grafana for novel applications like industrial IoT and business intelligence. Although the GAP is currently closed, interested parties are encouraged to express interest for future opportunities, highlighting Grafana Labs' commitment to nurturing a vibrant global community of developers.

Grafana Labs, known for its contributions to the open-source monitoring ecosystem, announced its transition from Cortex to launching Grafana Mimir, a scalable long-term storage solution for Prometheus, capable of handling over a billion metrics. At the upcoming virtual KubeCon + CloudNativeCon EU event, Grafana Labs team members will participate in various sessions, including talks on optimizing Prometheus's scalability and performance, addressing memory usage issues, and introducing the OpenMetrics format for standardized metric exposition. The event will also feature discussions on enhancing Prometheus's histogram functionality and will include a Prometheus booth staffed by Grafana Labs experts. The conference promises insights into the latest developments in Prometheus and related technologies, with opportunities for attendees to engage with maintainers and industry experts.

The Loki 1.6.0 release introduces significant improvements and new features, enhancing both performance and functionality across its components. Metric query performance has been optimized, achieving speeds up to 10 times faster, thanks to a refactor that separates log and metric query processing. The query frontend benefits from parallel query processing and sharding, improving efficiency. Enhancements to the query language include new functions and operators, while Promtail now supports the Loki Push API, allowing logs to be pushed from any client. The Promtail Pipeline gains stages for replacing sensitive data and dropping log lines based on specific criteria. Other notable updates include improved error messages, additional configuration flags, and increased data integrity checks from Loki Canary. The LogCLI tool now offers color coding for log lines and automatic batching of queries, ensuring more detailed and efficient log analysis. Overall, the release includes 187 pull requests, reflecting the ongoing efforts to enhance Loki's capabilities and user experience.

Grafana Labs, after ceasing contributions to Cortex, has launched Grafana Mimir as an open-source long-term storage for Prometheus, enabling scalability to over 1 billion metrics. This blog post explores the challenges and solutions encountered in scaling Cortex's blocks storage, such as inefficiencies in querying non-compacted blocks and the need for vertical and horizontal compaction to optimize query performance. The Cortex architecture uses a hash ring for sharding and replication, allowing for dynamic scaling and efficient distribution of workloads across store-gateways. The compactor's single-threaded nature poses scalability limitations, which are mitigated by per-tenant sharding and other optimizations. Future improvements will focus on further query optimizations and shuffle sharding to enhance the system's scalability. The work represents a collaborative effort within the Cortex and Prometheus communities, highlighting Grafana Labs' ongoing commitment to improving metrics storage solutions for large-scale deployments.

Grafana 7.1 introduces native support for InfluxDB's Flux query language, enabling users to gain enriched insights from time series datasets by combining multiple data sources. This integration facilitates complex queries and allows for real-time data correlation across different data types, enhancing observability and eliminating inefficiencies associated with siloed data. Flux, which offers a minimal learning curve, supports advanced features such as joins, math functions across measurements, and integration with various databases, allowing users to create sophisticated and enriched visualizations. Enterprises like IBM utilize this integration to monitor and optimize performance data in large-scale production environments, demonstrating the practical application of Grafana and InfluxDB's capabilities. The update allows users to transition smoothly from InfluxQL to Flux, while Grafana's built-in query builder makes it straightforward to create powerful dashboard charts.

Grafana Labs and New Relic have announced a partnership to enhance the integration of their observability tools, emphasizing the importance of being database-agnostic to provide a comprehensive view of data from various sources. Grafana will now serve as a frontend for New Relic's Telemetry Data Platform, allowing open-source Grafana users to utilize built-in support for PromQL and connect with New Relic, while Grafana Enterprise customers benefit from a native NRQL plugin. This collaboration highlights the growing prominence of Prometheus, with Grafana as a key visualization tool and New Relic offering robust support for Prometheus as a backend, thereby expanding developers' options for managing their data. The partnership aims to strengthen both communities by leveraging the strengths of both platforms and offering features like scalability and security tailored for large organizations, along with a trial period for Grafana Enterprise.

Grafana k6's new v0.27.0 release introduces an enhanced execution engine and a scenarios API, allowing users to configure and model load on systems under test with various executors, including the constant-arrival-rate executor. This executor enables generating a constant request rate by dynamically adjusting active virtual users (VUs) to maintain the desired iterations per time unit. The article explains configuring scenarios using this executor, detailing terms like rate, timeUnit, duration, preAllocatedVUs, and maxVUs, and emphasizes the importance of considering performance issues and realistic expectations when setting up tests. The new approach eliminates the need for previous JavaScript workarounds, simplifying the process of achieving a fixed requests per second (RPS) rate, and offers improved support over earlier versions.

Torkel Ödegaard, the creator of Grafana, hosted an "Ask Me Anything" (AMA) webinar to engage with the Grafana community, answer questions, and reduce the mean time to question answered (MTTQA) for users. He shared insights into the design philosophy behind Grafana, emphasizing a clean, distraction-free interface to enhance data visibility. The webinar also addressed customization options available in Grafana Enterprise, such as white labeling to personalize the user interface. As Grafana has scaled to support numerous data sources and use cases, the company faces challenges in managing community support and issue triage, while also encouraging developer contributions. Efforts include creating developer guides and fostering community engagement to address these scaling challenges effectively.

In a detailed tutorial, the process of sending logs from Amazon's Elastic Container Service (ECS) to Loki, a log aggregation system from Grafana Labs, is explored, aiming to enhance performance and reduce costs. The guide outlines using AWS's log router, Firelens, to forward logs and metadata from workloads to a Loki instance, enabling consolidated log queries through Grafana. It details the setup of an ECS cluster using Fargate, the creation of a task definition involving a Firelens log router, and a sample application to generate logs. The tutorial advises configuring a task definition's log driver to use awsfirelens for log forwarding and includes steps for creating a service to run tasks, emphasizing the need for proper subnet and security group configuration to ensure connectivity and log visibility. Additionally, it highlights the use of log labels to filter and explore logs within Grafana, suggesting further exploration of the LogQL query language for more advanced log analysis.

The upcoming webinar, scheduled for August 6, aims to introduce participants to the concept of distributed tracing with a focus on using the open-source tools Jaeger, Loki, and Grafana. Led by Joe Elliott, a Senior Backend Developer at Grafana Labs, the session will cover the fundamentals of tracing, the questions it can answer, and common open-source solutions for tracing. Participants will learn how to instrument a simple application for tracing, offload traces to a Jaeger backend, and utilize Grafana to visualize the traces, along with performing a typical Loki search. Interested individuals can register to attend the live session or receive a recording for later viewing.

As engineering teams increasingly adopt microservice-based architectures, traditional testing strategies often fall short in addressing the dynamic environments of distributed systems. To tackle this, Netflix pioneered Chaos Engineering with the development of Chaos Monkey, a tool that deliberately introduces system failures to identify architectural weaknesses and improve system reliability. This methodology has spurred the creation of various tools, such as Gremlin, which allows safe experimentation on complex systems and is used by companies like Expedia and Mailchimp. Effective monitoring is essential in Chaos Engineering, with Grafana being a popular tool among Gremlin users for visualizing system impacts during experiments. To enhance understanding and implementation of Chaos Engineering, a webinar featuring experts from Gremlin and Grafana is scheduled to offer insights into improving incident response through better monitoring and testing practices.

The post outlines the process of streaming Graphite metrics to Grafana Cloud using carbon-relay-ng, a Go-based tool for aggregating, filtering, and routing metrics. It provides a step-by-step guide on setting up a Grafana Cloud instance, generating an API key, and configuring carbon-relay-ng to publish metrics. The configuration process includes setting up the carbon-relay-ng.conf file, ensuring correct user credentials and API key integration, and utilizing environment variables for security. It emphasizes the importance of matching storage schemas and highlights the capability to manage multi-instance and high-availability setups for scalability. Once configured, metrics can be queried through a Grafana Cloud instance, and users can create or import dashboards and set alerting rules. The post suggests importing a Carbon-relay-ng dashboard to monitor the tool's performance and directs users to further configuration documentation for advanced metric handling and alternative data destinations.

Agnès Toulet is a Grafana Enterprise software engineer based in Paris, France, who joined Grafana Labs for its company culture and to venture into the open-source world. As a full-stack developer, she contributes primarily to Grafana while also engaging in personal projects like a Unity game created for a Ludum Dare game jam. In her leisure time, Agnès enjoys reading and writing fantasy and science fiction, drawing, 3D modeling, playing video games, running, and cooking, particularly pastries. For work concentration, she listens to music from bands like Imagine Dragons and Maroon 5. Her favorite food is ice cream, and she recently binge-watched iZombie on Netflix. Agnès wishes for the superpower of teleportation to simplify travel.