July 2019 Summaries

Public Grafana dashboards offer intriguing insights into how organizations visualize their data for various purposes, ranging from scientific research to smart building analytics. Fermilab, a U.S. Department of Energy national laboratory, uses Grafana to display resource distribution for its HEPCloud platform, which supports particle physics experiments with computing resources. The Open Science Grid, a U.S. distributed computing partnership, showcases dashboards illustrating the diverse scientific fields it supports with over a billion CPU hours annually. In Mumbai, the Smart Energy Informatics Lab uses Grafana to present real-time data on power consumption in smart classrooms, while Stockholm-based catalystBIT demonstrates building analytics through Grafana dashboards that aggregate data from IoT sensors in commercial real estate. This roundup encourages sharing of unique dashboards on Twitter for potential inclusion in future showcases.

Attending tech conferences can significantly advance a woman's career, especially in male-dominated fields, by offering networking opportunities, exposure to new ideas, and a platform to amplify women's voices. The author's experience at the /dev/color In Motion conference highlights the importance of representation and community support, noting that seeing Black executives and participating in women-led sessions provided a sense of belonging and inspiration. Conferences like /dev/color not only offer technical learning opportunities but also emphasize soft skills, such as negotiation and personal development, which are crucial for career advancement. The author underscores the value of community-based organizations like /dev/color in empowering underrepresented groups and providing ongoing support through local chapters, monthly meetings, and interactive discussions. This approach helps members to navigate their careers more effectively, gain fresh perspectives, and foster a supportive professional network.

Sofia Papagiannaki, a backend engineer at Grafana Labs, is based in Heraklion, Crete, and focuses on software development. Although she has not been as active in open-source projects as she would like, her recent joining of Grafana has revitalized her contributions on GitHub. In her leisure time, Sofia enjoys disconnecting from the digital world to read contemporary and classic literature, attend weekly local cinema screenings, and participate in social and ecological initiatives. She prefers listening to music while coding, choosing different genres for various tasks, and uses spaces for indentation in her code.

Loki, launched at KubeCon North America in December and inspired by Prometheus, is a service designed for efficient storage, search, and aggregation of logs, with seamless integration into Grafana. It is adaptable to both microservices and monolithic architectures, correlating logs and metrics to help users save costs. One of its most requested features involves the manipulation of log lines, allowing for operations such as extracting labels and metrics, setting timestamps, and modifying log entries before they are sent to Loki. This functionality is facilitated by a log processing pipeline that utilizes stages such as JSON and regex to extract and manipulate log data, improving query performance through the use of labels. The pipeline is designed to be flexible, allowing for the addition of new stages to accommodate diverse use cases. Future developments may include capabilities for splitting log lines and sharing pipeline configurations to enhance usability and collaboration.

On July 19, Grafana Cloud experienced a 30-minute outage in its Hosted Prometheus service due to issues with Kubernetes Pod Priorities, particularly when a new Cortex cluster was deployed without updated priority configurations. The incident resulted from the preemption of production Ingesters due to an oversight in setting priority levels, leading to a cascading failure of the Cortex microservices architecture. Prompt alerts allowed engineers to diagnose and resolve the issue by scaling the Kubernetes cluster to accommodate both new and existing workloads. Despite the outage, no data was lost, and customers’ Prometheus servers effectively buffered and replayed writes. Grafana Labs outlined several measures to prevent future occurrences, including refining configuration processes, requiring design documents for significant changes, and automating proxy sizing to handle overloads. The incident also provided valuable insights into the automatic recovery capabilities of Cortex and the utility of Grafana Loki for log verification.

k6 is an open-source load testing tool developed by Load Impact, designed specifically for developers to handle performance testing with a focus on simplicity and scalability. Launched in February 2017, k6 allows developers to write test cases in JavaScript, making it accessible and familiar while avoiding the limitations of esoteric scripting languages or GUI-based tools. The tool integrates seamlessly into continuous integration and delivery pipelines, supporting automation. To handle time series metrics, k6 leverages InfluxDB, and for visualization, it uses Grafana, known for its flexible dashboards. The community has contributed various dashboards to enhance k6's usability. In addition to its open-source version, k6 offers a SaaS product, simplifying infrastructure and result storage management, akin to Grafana Labs' model. k6 aims to be the preferred tool for developers by ensuring performance stability amidst rapid development cycles.

Tobias Skarhed is a frontend engineer at Grafana Labs, working as a summer intern while studying to become a space engineer at Luleå University of Technology in Sweden. He contributes to open-source projects like Grafana and engages in personal hobby projects. In his free time, Tobias enjoys singing in a choir, hiking with friends, and recovering from a leg injury that has paused his running activities. He prefers coding with music and ensures minimal distractions to stay focused, occasionally missing meetings due to being deeply engrossed in his work. His favorite ice cream is vanilla with chocolate pieces, and he prefers tabs over spaces in coding, though he accommodates others by converting tabs to spaces. He recently binge-watched "Chernobyl" and recommends it. Tobias can be found on GitHub as dehrax and on Twitter as @tskarhed.

Loki, a Prometheus-inspired service for optimizing log storage, search, and aggregation, has been making strides since its launch at KubeCon North America. As it approaches general availability, Grafana Labs has been enhancing its functionalities, such as the loki-canary project, to ensure log integrity during rollouts. Loki-canary acts as an early detection system for missing logs by writing logs at set intervals, which are then monitored via websocket connections. It generates metrics to identify missing, out-of-order, or duplicated logs, alerting users through Prometheus. Initial deployments revealed that websocket disconnections during server rollouts could result in missed logs, prompting the addition of direct query functionality to verify logs' presence. With hundreds of digital "canaries" operating across clusters, Grafana Labs can effectively monitor Loki's performance, ensuring no logs are lost.

A recent inquiry from a customer about translating esoteric IDs or serial numbers into user-friendly names in Grafana led to an exploration of how this can be achieved using SQL, which Grafana now supports. The process involves storing data, such as service IDs and corresponding display names, in a data source accessible by Grafana, like MariaDB/MySQL or Postgres. Once the data is in the database, Grafana's Dashboard Settings allow for the creation of new variables, where the database column assigned to __text is used for display, and __value is used during queries. This approach enables the display of user-friendly names while using the original values for querying, and it works seamlessly across various dashboard elements, including dropdowns and graph titles. The article emphasizes attention to security in production environments and offers further guidance through the MySQL data source documentation, inviting additional inquiries about monitoring best practices.

Meta tags are being developed for Metrictank, a scalable Graphite solution, to efficiently manage large volumes of metrics without the overhead of storing redundant tags individually. Sponsored by Bloomberg, this feature allows users to add additional metadata, such as data center or operating system, to metrics for better filtering and grouping in Grafana/Metrictank while minimizing resource usage. Meta tags are treated as extrinsic attributes, similar to Prometheus's series joining, but offer a more seamless and transparent user experience. The implementation involves linking meta tags to "metarecords" and incorporates an enrichment step to make the tags appear as regular ones when querying data. Users can manage these tags via an API, and the system will operate on a relaxed, eventually-consistent model, gradually updating tag associations across the cluster. This development aims to provide a cost-effective and seamless way to handle extensive metric tagging needs, enhancing Metrictank's capability to support large-scale users efficiently.

Loki, a Prometheus-inspired logging service launched at KubeCon North America, is nearing general availability with significant developments aimed at optimizing log storage, search, and aggregation. Grafana Labs has introduced a Docker logging driver plugin and support for systemd to enhance Loki's functionality by extending the sources from which logs can be collected. The Docker driver allows logs from non-Kubernetes workloads to be sent directly to Loki without the need for complex configurations, making it particularly useful for ECS users transitioning from Splunk. Meanwhile, the systemd support enables Loki to pull systemd journal logs from legacy workloads, broadening the scope of logs accessible to users. These advancements underscore Loki's utility in correlating logs and metrics to streamline operations and reduce costs, with further features such as loki-canary and query optimization on the horizon.

Shavonn Brown is a developer on the plugins team at Grafana Labs, based in Atlanta, Georgia. She engages in both frontend and backend development but has limited experience contributing to open-source projects, despite attending an open-source conference and having an interest in the idea. Her GitHub handle is shavonn, and she is known as @DevWithClaws on Twitter. In her free time, she enjoys spending time with her family, watching streaming services, playing with her dog, and trying new experiences. When coding, Shavonn prefers using noise-canceling headphones to listen to music or audiobooks, or having Netflix on in the background, and she is flexible regarding the use of spaces or tabs, opting for whatever the team prefers.

Prometheus v2.11, released on July 9, introduces several enhancements and features, including the compression of WAL records using Snappy to significantly reduce their size, improved query performance, and the option to use Alertmanager API v2. This version enhances Kubernetes service discovery by adding support for initContainers, which can provide valuable troubleshooting information and help detect issues. Metric naming was updated, with prometheus_tsdb_wal_reader_corruption_errors renamed to prometheus_tsdb_wal_reader_corruption_errors_total, and a new metric, prometheus_http_requests_total, introduced to track HTTP requests to the Prometheus API. Performance improvements include reduced allocations in PromQL aggregations, optimized remote-write allocations, and enhanced query speed through the use of regexp matchers for set lookups. Additionally, a bug fix for unsafe snapshots was implemented through a collaborative effort, providing more robust snapshot functionality.

Prometheus v2.11, released on July 9, introduces several enhancements and new features following its adoption of a six-week release cycle post-graduation from CNCF. Key updates include an option to compress WAL records using Snappy, improvements in query performance, and a change in metric names, such as renaming prometheus_tsdb_wal_reader_corruption_errors to prometheus_tsdb_wal_reader_corruption_errors_total and adding a new metric prometheus_http_requests_total. The release also allows the use of Alertmanager API v2 and introduces Kubernetes service discovery for initContainers, which may provide valuable troubleshooting information. Performance enhancements include reduced allocations in PromQL aggregations, remote-write allocation improvements, and optimized query handling using regular expressions for set lookups, particularly benefitting Grafana dashboards. The update also addresses a bug related to unsafe snapshots. Users are encouraged to review the release notes for a comprehensive overview and to download the new version.

Grafana Labs' community of over 600 developers actively contributes to open-source projects and frequently engages in discussions about optimizing Grafana usage. The community addresses popular queries, such as transferring a Grafana database to a new server, which requires copying the grafana.db file and ensuring consistent grafana.ini settings and proper permissions. New users can be created in Grafana through the Server Admin under the Users tab. Another discussion involves extracting values from metrics where the data is stored as labels rather than metric values, with solutions like using the SingleStat panel in dashboards to display version information effectively. Grafana Labs emphasizes collaboration and invites users to participate in the community board for further discussions.

Prometheus, a prominent open-source monitoring system that graduated from the Cloud Native Computing Foundation in 2018, has seen a surge in development activity across its ecosystem, with regular releases and numerous enhancements being made. Key improvements include fixing longstanding bugs, optimizing PromQL for efficiency, and introducing subquery support and sample limit configurations. The project has also seen increased integration efforts with remote storage solutions such as Cortex and Thanos. Looking forward, Prometheus aims to adopt OpenMetrics standards, support server-side metric metadata, and enable backfilling time series data. Contributions from programs like Google Summer of Code have been instrumental in advancing these developments, underscoring the vibrant and collaborative nature of the Prometheus community.

Lazy loading of dashboard panels, a long-requested feature by the Grafana community, was implemented in version 6.2 to efficiently manage data queries and reduce load spikes on data source backends. Previously, Grafana would query all panels immediately upon opening a dashboard, even those not immediately visible, potentially causing unnecessary strain. With lazy loading enabled by default, only panels in view are queried, optimizing performance and encouraging users to strategically design dashboards by placing less frequently viewed, resource-intensive queries further down. Additionally, for React-based panels like Gauge and Bar Gauge, only visible panels will refresh when triggered, updating others as they scroll back into view. This feature aims to enhance user experience by reducing backend load and improving dashboard responsiveness.

Josue Abreu is a backend engineer originally from the Dominican Republic, now based in London, who works on the Enterprise offering at Grafana Labs. He is actively involved in the Grafana open-source project and has a consistent online presence under the handle "gotjosh" on GitHub and Twitter. In his personal life, Josue enjoys snowboarding and was once a competitive gamer, though he now prefers to watch professionals play. His work environment preferences have evolved from needing music while coding to preferring complete silence, and he recently enjoyed binge-watching the BBC show "Line of Duty," appreciating its writing and acting over high production value.

Grafana Labs employees share insights into their personalized home workspaces, revealing creative solutions and setup preferences that support their work in dashboard creation and service hosting. Many team members utilize multiple monitors with inventive mounting solutions, like Dan Cech’s custom Vesa adapter and Victor Cinaglia’s pursuit of stability with a Vivo arm. Cable management is a common theme, with employees employing zip ties, NeatLinks trays, and cable holders to keep their desks organized and clutter-free. Additionally, Ronald McCollam showcases a versatile, foldable desk design, while Simon Crute and Dominik Prokop discuss their adaptations for multi-monitor setups. These setups not only enhance productivity but also reflect the individuality and engineering creativity of the Grafana Labs team.

Michelle Tan's article highlights strategies to decrease Mean Time to Resolution (MTTR) and increase uptime using VictorOps and Grafana, as discussed by Melanie Postma at GrafanaCon 2019. Postma emphasizes the importance of robust monitoring solutions and effective alert management to ensure alerts reach the right personnel promptly, thereby minimizing downtime. VictorOps, an automated alerting system acquired by Splunk, is praised for its ability to reduce alert fatigue and enhance accountability, exemplified by PSCU's reduction of MTTR from four hours to two minutes. Postma also discusses the critical role of understanding the impact of deploys and providing detailed context in alerts, such as attaching Grafana graphs and runbooks, to expedite incident resolution. Furthermore, data-driven improvements and post-incident reviews are encouraged to continually enhance processes and prevent future incidents, fostering a culture of observability and proactive problem-solving.

Amgen utilizes Grafana to efficiently manage on-call rotations and mitigate burnout among engineers, leveraging the tool to track and visualize on-call data. By implementing dashboards, Amgen improves operations by facilitating quick connections between individuals seeking assistance and those available to provide it, particularly during off-hours or for external customers. This system allows for streamlined communication through features like SLO dashboards that include current on-call contacts and escalation levels. Additionally, Grafana aids in balancing workloads and enhancing team morale by enabling managers to monitor the number of on-call days and incidents each engineer handles, which empowers both managers and engineers to address imbalances and reduce stress. This data-driven approach not only optimizes operational efficiency but also fosters a more positive on-call experience, offering engineers the tools to discuss workload concerns with their managers effectively.