April 2020 Summaries

Cortex initially faced a significant flaw in its ingester service, which risked data loss if an ingester crashed, as it temporarily stored incoming series data in memory before writing it to long-term storage. To address this, Cortex integrated a Write-Ahead Log (WAL) similar to Prometheus' TSDB, allowing ingesters to log events to a file and replay them on crash recovery, thus restoring in-memory states. However, WAL alone was insufficient under heavy loads, prompting the use of checkpoints, which store compressed data chunks on disk for faster replay. Originally experimental, these enhancements have undergone thorough testing and improvements, such as optimized disk writes and a switch to Prometheus TSDB WAL format, and are poised to lose their experimental status. Those interested in adopting WAL with Cortex can refer to the production guide, while further insights into Cortex can be gleaned from a Grafana Labs webinar featuring key contributors.

Grafana 7.0 is set to launch soon, featuring a revamped table panel that addresses numerous user requests accumulated over time. This new panel allows for the inclusion of various visualizations within table cells, such as rendering values as bar gauges, thanks to a new panel architecture and style override system that promises future enhancements like sparklines and images. The beta version is currently available for download, and further announcements are anticipated as the release date approaches, particularly during the upcoming GrafanaCONline event, where interested users can sign up for updates and live streamed presentations.

As companies transition to remote work due to the COVID-19 pandemic, Grafana Labs shares insights from its remote-first culture to enhance productivity, especially through effective use of Slack. Key practices include configuring Slack to add new lines with the Enter key, using markup formatting for clearer communication, and setting reminders to avoid the pressure of real-time responses. Users can manage message priorities by marking them as unread, setting message reminders, or saving important messages, with saved items functioning as a personal to-do list. Channel management is optimized by categorizing them into starred, regular, and muted, allowing users to focus on critical discussions while reducing distractions. Additionally, sorting channels by priority rather than alphabetically enables users to address the most important messages first, ultimately encouraging a more intentional and organized approach to handling Slack communications.

Alice Farrell, the VP of People Operations at Grafana Labs, works remotely from North Yorkshire, UK, and has been with the company since January 2019, overseeing the recruitment and HR aspects of the growing organization. Since joining when the company had 37 employees, she has witnessed its expansion to over 140 staff members across 27 countries, which presents both challenges and rewards in her role. Alice describes her job as a pleasure due to her supportive team and the diverse, talented colleagues she interacts with daily. Outside of work, she manages life with two young boys and enjoys activities like cycling, reading, and socializing, although her free time is often consumed by parental duties and relaxation through TV shows like "The Marvelous Mrs. Maisel" and "The Good Place." Originating from the Highlands of Scotland, she retains a fondness for local heather honey and Scottish oatcakes.

Grafana v7.0, set to release next month, introduces a new auto grid layout feature for gauge and stat panels, enhancing their display options beyond the previous horizontal or vertical stacking by optimizing space usage dynamically. This update will allow for more efficient and visually appealing arrangements of sub-visualizations within a grid format. The 7.0 beta version is currently available for download, and additional announcements are expected during the upcoming GrafanaCONline event, where interested users can sign up for livestream updates and presentations.

GrafanaCONline is an online event scheduled from May 13 to 29, offering a series of free, flexible presentations to replace the in-person GrafanaCon initially planned for Amsterdam. This virtual conference allows attendees worldwide to access live-streamed sessions and a keynote by Grafana leaders like CEO Raj Dutt, Torkel Ödegaard, and Tom Wilkie, focusing on the latest in Grafana 7.0, Prometheus, Loki, and other open-source monitoring tools. The event promises over 15 sessions available on-demand, an online AMA with Grafana Labs product teams, and networking opportunities through a dedicated Slack channel, all aimed at providing valuable insights into the observability ecosystem. Registration is open to everyone, ensuring easy access to these resources and updates.

Ed Welch's piece on Loki delves into how labels enhance log queries, drawing parallels with Prometheus while catering to users unfamiliar with the latter. Labels in Loki are key-value pairs that define streams, simplifying the process by eliminating metric names and using streams instead of series. The piece emphasizes the importance of managing cardinality, as high cardinality can lead to inefficiencies. Loki's effectiveness stems from parallelization, breaking queries into smaller pieces for efficient processing. Best practices suggest using static labels and being cautious with dynamic ones to avoid creating numerous streams, which can degrade performance. The article also discusses the significance of proper log ordering and chunk management, highlighting that Loki's design favors fewer streams and a smaller index, optimizing for cost-effectiveness and performance through parallel processing. The narrative encourages a shift in mindset from traditional log storage solutions, advocating for strategic label use and filter expressions to leverage Loki's potential.

Machine learning company ML6 collaborated with Accolade Wines to reduce waste in their bottling operations by using Grafana to create a real-time monitoring dashboard. Accolade Wines, one of the largest wine producers globally, faced challenges in pinpointing inefficiencies due to disparate monitoring systems from different equipment manufacturers. By integrating data from various stages of the bottling process and visualizing it through Grafana, ML6 enabled operators to identify and address wine losses immediately, resulting in a 200% ROI within six months and reducing financial losses by £75,000. The project has not only improved operational efficiency but also laid the groundwork for future advancements using machine learning to detect previously unknown sources of loss, with plans to replicate this solution for other manufacturers and explore applications in the life sciences sector amid the COVID-19 pandemic.

The guide discusses using Postman in conjunction with the k6 load testing tool to efficiently test APIs by converting Postman collections into k6 scripts. It explains the process of using the postman-to-k6 converter to transform a Postman collection into a k6 script, which can then be used to perform load tests on APIs, highlighting the capabilities and limitations of both tools. The text provides a step-by-step approach for installing necessary tools, converting collections, and running tests, emphasizing the strengths of using k6 for load testing compared to Postman’s focus on API functional testing. It also covers the metrics and features supported by k6, such as WebSocket testing, while noting the importance of rewriting scripts for better maintainability after the initial conversion. The article aims to facilitate onboarding to the k6 platform, promoting its integration capabilities and encouraging users to utilize Grafana Cloud for enhanced performance testing.

Jacqui Jeanis, a Senior Enterprise Account Executive at Grafana Labs, plays a pivotal role in helping organizations implement solutions by providing necessary resources and has been recognized for her impact with the 2019 Maximum Impact award. An "Original Grafanista," Jacqui joined the company over 2.5 years ago, initially connecting with the team in 2016 at a Grafana meetup discussing Prometheus, a project that has grown significantly alongside Grafana. Outside of work, she enjoys boxing with world champion trainers in NYC, which has taught her dedication and focus, and values her intuitive "superpower." Jacqui finds immense satisfaction working with her supportive and driven team and recently fulfilled a personal dream by visiting the Amalfi Coast, which she described as a "warm fuzzy dream."

Richard "RichiH" Hartmann's piece explores the art of asking questions effectively within various contexts, emphasizing the importance of respect, preparation, and clarity. He argues that asking questions is a form of trading time, where the questioner seeks to save their own time by utilizing someone else's expertise. Hartmann underscores the necessity of making an honest attempt to solve one's problem before reaching out for help, which not only shows respect for the respondent's time but also often leads to self-discovery and solutions. He advocates for providing clear context and stating one's goals rather than just strategies, which allows others to assist more effectively. Additionally, the article highlights the value of signaling appreciation and thankfulness, fostering a respectful and pleasant interaction that can encourage future engagement. Hartmann's unique approach focuses on the broader context of the question rather than assuming the reader is uninformed, promoting a more constructive and respectful dialogue.

Grafana has enhanced its integration with Loki, allowing users to easily use Loki as a metric datasource in dashboards, eliminating the need for previous hacks that involved adding Loki as a Prometheus datasource. This improvement, available from Grafana v6.6, enables users to create dashboards that combine log data with metrics without requiring any code changes for metric instrumentation when the application lacks it. Users need to load data into Loki, choose a visualization type such as Graph or Gauge, and write a metric query using Loki's LogQL, with options for fixed intervals or using the Grafana variable $__interval. The seamless integration allows for the creation of application dashboards based solely on logs, and an upcoming webinar on April 22 will cover the basics of Loki, its setup, and usage within Grafana, along with an introduction to querying.

Histograms have significantly impacted the monitoring of time series in Prometheus by addressing limitations of earlier methods like summaries with precalculated quantiles, which were inflexible and couldn't be aggregated across various dimensions. Björn Rabenstein's trilogy of talks, including presentations at PromCon EU 2019 and FOSDEM 2020, explores the evolution and challenges of histograms within Prometheus. While histograms resolved many issues inherent in earlier designs, some challenges remain due to historical design decisions that, despite creating issues for histograms, contributed to Prometheus's overall success. The talks also highlight the influence of Prometheus's text format, improvements in its v2 release, and standardization efforts like OpenMetrics. Rabenstein plans to present further advancements in histograms at the postponed KubeCon + CloudNativeCon Europe 2020, promising additional insights from ongoing research.

The blog post by Eldin Nikocevic and Christine Wang discusses the use of the Datadog plugin for Grafana, highlighting its role in enhancing operational insights by integrating Datadog metrics with other data sources like Prometheus and Grafana Loki. It details the process of setting up Datadog as a data source in Grafana, emphasizing features such as the Cache Interval to manage API rate limits, and offering tips on using the Datadog query editor and advanced functions. The post also explains how to perform arithmetic queries, use templating for metrics and tags, and leverage ad-hoc filters for dynamic data exploration. It concludes by inviting users to explore the Datadog enterprise plugin available for Grafana Cloud or Enterprise license holders, with a call to action to engage with the authors on Twitter and try Grafana Cloud Pro through a free trial.

Dave Russell, the Vice President of Solutions Engineering at Grafana Labs, discusses his role and personal interests in a team profile. Based in rural Herefordshire, UK, Dave leads a global team that bridges the gap between Grafana's commercial, engineering teams, and its clients, focusing on technical challenges and solutions. He enjoys podcasting about big tech and enterprise topics, citing the Roaring Elephant Podcast as one of his long-standing projects. In his leisure time, Dave indulges in a variety of interests, from operating a mini digger to balancing items in random places. He also expresses a fondness for music as a work motivator and shares a love for Italian cuisine and the TV series "Band of Brothers." Additionally, he reflects on his aspirations for superpowers like pausing time and creatively locating objects, while also recounting his latest bucket list achievement of acquiring a property with numerous renovation projects.

The shift to remote work due to the COVID-19 pandemic has transformed the way workshops are conducted, with many companies like Grafana Labs, which already had a remote-first culture, adapting by sharing strategies and tips for effective virtual collaboration. Remote workshops differ significantly from traditional in-person ones, lacking real-life interactions such as sticky notes and energizers, but they can be successful with the right digital tools and methods, like collaborative text editors and digital whiteboards. Important aspects of conducting remote workshops include the use of collaborative tools to replace physical interactions, timeboxing to prevent fatigue, careful management of the mute function to maintain engagement, and incorporating energizers to boost creativity and team spirit. Mixing in-person and remote groups can present challenges, so it's often better to conduct fully remote sessions to ensure equal participation. The author emphasizes the importance of creativity, adaptability, and sharing experiences online to build a collective resource for remote work best practices, which can help navigate current and future crises while fostering a more remote-friendly world.

Eldin Nikocevic and Ronald from Grafana Labs provide a detailed guide on creating a Grafana dashboard using Prometheus and Loki for efficient log searching. Inspired by the ad-hoc querying capabilities of Grafana 6.0's Explore feature, they designed a user-friendly interface that simplifies the process of searching logs without needing to write LogQL queries. The dashboard utilizes Prometheus variables and template variables, including a crucial "text box" variable, to filter and search logs effectively. This innovative setup is particularly useful for troubleshooting and incident response, offering a streamlined way to access log data quickly. The process involved combining Prometheus variables to select the search cluster and pod, ultimately enabling searches through Loki with specific queries. They also mention enhancements in Loki v1.4.0 and invite users to join an upcoming webinar for a deeper understanding of Loki's functionalities.

On March 16, Grafana Cloud's Hosted Prometheus service encountered a 12-minute partial outage in the London region, caused by a misconfigured etcd client. The outage, which delayed data storage but did not result in data loss, was triggered when an etcd leader node was terminated during a scheduled Kubernetes upgrade, causing TCP connection issues with the Cortex distributor responsible for deduplicating samples. The problem was detected through SLO-based alerting and resolved by restarting the affected distributors, while ensuring the future prevention of similar incidents through better configuration of gRPC keepalive probes and enhancements in the Kubernetes upgrade process. The incident highlighted the effectiveness of Grafana Loki in facilitating swift log analysis and recovery, underscoring the importance of robust monitoring and alerting systems in maintaining service reliability.

Grafana Cloud and Enterprise are favored by various companies for their ability to streamline data visualization and improve operational efficiency across diverse industries. A New York-based bank harnesses Grafana to consolidate data from multiple platforms, enhancing developer productivity by saving significant time through role-based dashboards. Another company utilizes Grafana to expedite their Mean Time to Innocence (MTTI) by visualizing real-time data from logging and APM sources without needing to merge them, thus avoiding delays and additional costs. A large gaming company turned to Grafana for a unified data view while transitioning to Kubernetes and Prometheus, leading to reduced costs, improved query performance, and increased developer productivity. Additionally, a major consulting firm adopted Grafana's hosted Prometheus/Cortex to gain better insights into their cloud infrastructure, significantly improving their incident response times and fostering wider adoption of these technologies within the company.

Matt Mendick, an engineering director at Grafana Labs, manages a team of backend engineers focused on open-source projects like Cortex, Loki, and Prometheus. After leading the development of Cortex for five years, Grafana Labs has shifted its focus to Grafana Mimir, a scalable long-term storage solution for Prometheus. Matt, based in Rochester, New York, values the opportunity to work in an open-source-oriented company and has contributed to internal processes and minor public contributions, such as fixing a logo link in Cortex's readme. Outside of work, he enjoys family time, preparing dinners, and attending CrossFit sessions. To enhance his work environment, he uses a high-quality microphone for video calls, and he recently binge-watched the series Silicon Valley.

Loki v1.4.0, a release featuring more than 130 pull requests from over 40 contributors, introduces significant enhancements including query statistics, regex optimizations, and improved client retry mechanisms for rate-limited failures. The update brings a new query frontend for performance gains through query parallelization, which is now measurable with enhanced statistics exposed via the API, allowing users to see data processing metrics. Regex filters have seen optimization, offering up to 300x speed improvements in some cases. Promtail can now perform dry runs to facilitate debugging, and retries are implemented for clients encountering 429 errors, providing resilience against server-side issues. Additional features include binary operators in LogQL, a metric stage for line counting, and configurable max_chunk_age to manage memory usage and log loss risks. The release also highlights the importance of upgrade considerations, especially for users of Loki versions 1.2.0 or older, and invites users to a webinar for a deeper understanding of Loki’s functionality.