June 2020 Summaries

In the context of a remote-first company, offsites are considered essential for fostering team efficiency, building trust, and facilitating direct human interaction, typically involving travel to a pleasant location to focus on strategic discussions. However, the COVID-19 pandemic necessitated the shift to virtual offsites, as demonstrated by Grafana Labs' recent experience. The virtual format offered benefits like reduced travel stress and environmental impact, as well as increased flexibility in participation, but also faced challenges such as the loss of natural group dynamics and social interaction due to the limitations of video platforms. Innovative solutions like using Zoom for video and Slack for discussion threads improved communication, allowing for efficient Q&A management. Despite the positive feedback and the potential for integrating virtual elements into future physical offsites, the lack of in-person social experiences was a notable drawback. The company plans to continue refining both virtual and physical offsites, considering a balanced mix of the two annually to maximize benefits while minimizing environmental impact and logistical complexities.

Grafana Labs' success in creating a supportive and happy workplace, as evidenced by its inclusion in Inc.’s 2020 list of Best Workplaces, is attributed to its strategic focus on people operations, remote-first culture, and intentional company culture. Alice Farrell, VP of People Ops, highlights the importance of early investment in people operations to define company values, recruit talent, and maintain low turnover rates. This approach has facilitated Grafana Labs' rapid growth from a small startup to a global company with over 150 team members. The remote-first model enables the company to recruit talent globally, enhancing diversity and flexibility, while a strong company culture and clear progression frameworks support team engagement and retention. Additionally, the organization's commitment to transparency, empowerment, and work-life balance contributes to its dynamic and inclusive environment, ensuring the sustained happiness and productivity of its employees.

Ganesh Vernekar is a software engineer at Grafana Labs, contributing primarily to Prometheus and Cortex open-source software projects, with occasional work on Thanos. Based in Dharwad, India, Ganesh joined Grafana Labs due to its promising growth and significant investment in Prometheus, aligning with his interests. He has recently worked on memory-mapping head chunks and improving snapshotting in Prometheus, as well as optimizing Cortex's new block storage. In his free time, Ganesh enjoys photography, watching movies, reading, researching interesting topics, and traveling. To focus during work, he listens to loud music, particularly EDM, and appreciates the flexibility, global connectivity, and challenging problems his job offers. His GitHub handle is "codesome" and Twitter handle is "@_codesome," and he recently binge-watched the Harry Potter film series.

Prometheus JMX Exporter enables monitoring of Java applications by bridging the gap between Java's native metric collection system, Java Management Extensions (JMX), and Prometheus, which is popular for collecting and querying metrics in cloud-native environments. The JMX Exporter converts JMX metrics into a format that Prometheus can understand, facilitating the integration of Java applications like Kafka into a modern monitoring setup. The tutorial demonstrates building the JMX Exporter using Maven, deploying Kafka, and incorporating its metrics into Grafana for visualization and alerting. The process involves setting up a Java environment, configuring the JMX Exporter as a Java agent, and ensuring the metrics are exposed on a specified port, with the option to compare Java's native jconsole output with that of Prometheus and Grafana. Once configured, this setup allows for effective monitoring of Java applications using modern tools.

Grafana 7.0 introduces a new React-based plugin platform that simplifies and accelerates the development of custom plugins, such as a scatter plot visualization. This tutorial guides users through creating a Grafana panel plugin using ReactJS and TypeScript, with a focus on visualizing scatter plots—graphs that map two variables as points on horizontal and vertical axes. The tutorial employs SVG for creating vector graphics and D3.js for transforming data, illustrating how to use these tools to generate scales and axes for the scatter plot. It highlights the importance of rendering data-driven SVG elements using React components and D3 for data transformations while managing the interplay between React and D3 for dynamic DOM updates. The tutorial also covers configuring Grafana to recognize the plugin, using Docker for development, and handling graphical transformations to position axes correctly. The article concludes by suggesting further enhancements, such as using actual data sources and adding user configurability to the plugin.

Emil Tullstedt's post provides a detailed guide on visualizing Prometheus histograms using Grafana, specifically tailored for users familiar with these tools. The post uses a fictional image hosting service's histogram data to demonstrate various queries in Grafana 7.0, including how to determine the number of files uploaded below certain sizes, calculate proportions, and visualize data distribution over time. It emphasizes the cumulative nature of Prometheus buckets and offers instructions on configuring Grafana panels, such as using bar gauges and heatmaps, to accurately display histogram data. The guide also highlights the importance of configuring Grafana settings to ensure numerical and temporal data are appropriately represented, and encourages users to share their histogram visualizations with the Grafana community.

Grafana's migration from AngularJS to React, particularly focusing on the templating system, was a complex endeavor that involved several strategies and challenges. Initially, the migration process faced difficulties with state mutations and polymorphism in the old system, leading to attempts like the proxy and top-down approaches that ultimately proved inadequate. A successful transition was achieved through a feature toggle approach, allowing for the separation of the new React/Redux system and facilitating the migration of UI components. This shift introduced architectural benefits, such as a clear separation between UI and state, while also presenting new challenges, such as testing thunks and managing plugin communication. Despite these hurdles, the transition improved code quality and facilitated future developments, leading to a more maintainable and scalable system.

Tracing is an essential tool for developers and engineers to gain insights into the performance of distributed applications, and Grafana Labs utilizes frameworks like Jaeger and OpenTelemetry for this purpose. While traditional head-based sampling methods often miss important traces due to their probabilistic nature, tail-based sampling evaluates complete trace data to make more informed decisions. However, the challenge arises when the OpenTelemetry Collector is scaled horizontally, leading to partial trace sampling and incomplete data. Grafana Labs addresses this issue by introducing the aggregate processor, which ensures all spans of a trace are sent to the same collector instance, allowing for accurate tail-based sampling. This processor uses a combination of fnv hash and jumphash to effectively distribute spans and minimize data loss during scaling. The implementation has proven successful in Grafana Labs' internal infrastructure, maintaining consistent trace sampling even during traffic spikes, and the processor is available for public use through their GitHub repository and Docker image.

Ivana Huckova shares her experience of creating an IoT project to monitor a sourdough starter using Grafana and various hardware components during the coronavirus lockdown. Inspired by Christine Sunu’s sourd.io, Huckova's project employs sensors to measure temperature, humidity, and the rise of the sourdough starter, collecting data with an ESP32 development board. The data is stored in an InfluxDB time series database and visualized in Grafana, allowing Huckova to track her starter's condition and potentially improve the quality of her bread. This project, designed to appeal to beginners in IoT development, includes detailed instructions for assembling the hardware and configuring the software, aiming to inspire others to explore similar creative projects.

Marcus Olsson introduces the Hourly Heatmap plugin for Grafana, which adapts the popular but no longer maintained Carpet plot plugin to Grafana's new React-based platform introduced in version 7.0. Unlike Grafana's built-in heatmap panel, the Hourly Heatmap panel plots hours of the day along the Y-axis, making it particularly useful for monitoring systems with continuous daily activity by grouping data into hourly buckets and mapping them to color spectrums to indicate traffic levels. This plugin retains the original functionality of the Carpet plot while adding community-requested features such as time zone customization and the ability to display data only during specified working hours. It also supports new field options, allowing multiple queries within the same panel and customization of elements like color palettes. Olsson advises users to replace the Carpet plot with the Hourly Heatmap for Grafana 7.0 with minimal effort but recommends testing it alongside the original to identify any potential issues.

Grafana Labs has enhanced the user experience for those working with Prometheus by introducing metric metadata directly in Grafana's Explore view, starting from Prometheus v2.15+ and Grafana v6.6+. This feature simplifies understanding and managing metrics by displaying metadata such as Type, Help, and Unit in the AutoComplete panel when typing a metric name, alleviating the need to delve into code for definitions. This improvement addresses common challenges like distinguishing between similar metric names and understanding metric types, which otherwise disrupt workflows. While Prometheus's Scrape Targets Metadata API endpoint previously allowed access to metadata, it often provided more information than necessary, making it difficult to locate relevant data. The integration of this metadata in Grafana's interface is aimed at enhancing user efficiency and clarity, with the support also extended to Cortex v1.1+ when paired with the Grafana Agent. Users are encouraged to provide feedback on this feature through the #cortex channel on Slack, and further information on Prometheus can be explored via an on-demand GrafanaCONline session.

Grafana 7.0 introduces a significant enhancement with the implementation of data transformations, expanding their availability beyond just the Table panel to all visualizations that support queries. Transformations in Grafana allow users to modify data before visualization by filtering, organizing, combining, performing reduce calculations, and executing simple math operations on time series data. Users can chain multiple transformations for sequential processing, with each step building on the previous one. This feature is particularly beneficial for optimizing dashboard performance, especially with shared queries, legacy data sources, and data sources lacking aggregation support. Transformations can be accessed through the Transform tab in the panel editor, reflecting efforts to standardize and unify data processing with a new data pipeline architecture centered on data frames. The update aims to enhance user experience by allowing more flexible and efficient data manipulation, inviting feedback and innovative applications from the user community.

Prometheus v2.19.0 introduces a significant feature that involves memory-mapping full chunks of the head block from disk, which reduces memory usage by 20-40% and accelerates restarts. This new feature benefits not only Prometheus but also other projects in its ecosystem, such as Cortex and Thanos, which utilize the Prometheus TSDB. The head block, storing the last 1–3 hours of data, now uses memory-mapping to load data into memory as needed, allowing for decreased memory allocation under certain conditions. Benchmark tests demonstrated that the reduction in memory usage varies with the churn rate and scrape interval; notably, lower churn and shorter scrape intervals result in greater memory savings. While this enhancement allows for more efficient resource allocation, it is crucial to plan for potential memory spikes during heavy queries. Additionally, during the replay of the Write Ahead Log (WAL), memory-mapping enables the omission of samples already covered by mapped chunks, reducing replay time by 20-30%.

The blog post by Marcus Olsson provides an instructional overview on how to use Grafana's dynamic dashboard features, focusing on the creation of automatically repeating rows and panels. It explains how template variables can be used to dynamically update dashboard content, allowing users to filter and display specific data metrics by selecting different instances or services from a drop-down menu. The guide covers the process of setting up custom variables for both panels and rows, illustrating how to repeat these elements based on variable selections to streamline the management of monitoring dashboards. It also highlights the potential pitfalls of overusing repeated panels due to increased query costs, advising users to apply these features judiciously. The post is part of the ongoing "Learn Grafana" series, and it encourages readers to explore additional resources for further enhancing their Grafana skills.

Loki, a log aggregation system developed by Grafana Labs, faced a performance bottleneck due to the slow regex package in Go, which hindered efficient log searching. The Go regex package, while prioritizing security against ReDoS attacks, was not performant for high-volume log filtering in Loki. To address this, the team implemented a workaround by leveraging the syntax sub-package that allows regex simplification, effectively using byte comparisons instead of complex regex operations. This approach significantly improved query performance, reducing execution times from 11.5 seconds to 1.5 seconds for certain queries, with overall speed enhancements ranging from 5x to over 300x. The improvement maintains the original security features of the Go regex package, as it does not replace it but optimizes its use. The team plans to expand this solution into a package for broader use and presented future plans for Loki at GrafanaCONline 2020, including enhancements to the LogQL query language.

Zach Choquette, a Talent Acquisition Manager at Grafana Labs, thrives in the dynamic environment of a growth-stage startup and is passionate about recruiting, with a focus on finding candidates who exhibit grit, intellectual curiosity, rigor, teamwork, ownership, polish, and a desire for impact. He values the unique culture at Grafana Labs, which combines a compelling mission, innovative products, and a remote-first work environment. Zach's hiring philosophy emphasizes collaboration and constant refinement, inspired by the idea of treating the recruiting process as both a sword and shield. Outside of work, he finds peace and excitement skiing, an activity that led him to relocate from Boston to Denver. He enjoys connecting with new opportunities and helping people advance in their careers, and his personal achievements include self-financing his college education and completing a challenging hike along the Kalalau Trail in Hawaii. Zach also enjoys playing basketball, watching movies, and experimenting with voice impressions.

Hiya, a company specializing in caller identity and spam blocking services, transitioned to Grafana Cloud to better manage costs and gain more control over their metrics. Initially using a vendor product, Hiya's team faced challenges with maintaining costs and lacked the ability to filter or aggregate metrics. The switch to Grafana Cloud Hosted Prometheus offered Hiya the industry-standard combination for monitoring Kubernetes clusters, providing the desired control and transparency over metrics. Grafana Cloud, powered by open-source Cortex, allowed Hiya to handle massive scale without managing infrastructure. Implementation involved migrating dashboards and adapting to Prometheus' query language, requiring training and support from Grafana Labs. Ultimately, the move resulted in significant cost savings, enhanced auditing capabilities, and a reduction in metrics, with Hiya's engineering teams benefiting from improved dashboards and operational insights. Looking forward, Hiya plans to integrate standardized dashboards and leverage new visualization features in Grafana, solidifying their satisfaction with the transition.

Grafana released versions 6.7.4 and 7.0.2 to address a critical security vulnerability affecting all versions from 3.0.1 to 7.0.1, specifically an SSRF vulnerability that does not require authentication, allowing unauthorized users to make HTTP requests via Grafana and potentially exposing network information. This issue, which was reported on May 14, 2020, and assigned CVE-2020-13379, can also lead to denial-of-service attacks by causing segmentation faults with invalid URL objects. To mitigate the vulnerability, users are advised to upgrade to the patched versions or block access to the vulnerable avatar feature, with Grafana Cloud instances and Enterprise customers receiving updates under embargo. Grafana encourages users to report security vulnerabilities via their designated email, offering a PGP key for encrypted communication, and maintains a Security Announcements section for updates and remediation details.

GrafanaCONline concluded after three weeks of engaging sessions, highlighting Grafana 7.0, Prometheus advancements, and the future of the Loki project, among other topics. The event featured a variety of discussions, from using Grafana in industrial process monitoring and reducing wine waste with machine learning, to innovative applications such as improving driving efficiency with a Raspberry Pi and enhancing Kubernetes clusters. Grafana Labs emphasized community involvement, encouraging contributions to their open-source documentation, while co-founder Raj Dutt and VP Alice Farrell discussed the company's transparency and culture. All sessions are available on demand, allowing attendees to revisit key insights shared during the conference.