April 2019 Summaries

Grafana has released versions 5.4.4 and 6.1.6, which include critical security fixes for installations between versions 5.4.0 and 6.1.6, addressing a vulnerability (CVE-2018-19039) that could allow users with Editor or Admin permissions to access any file readable by the Grafana process. The vulnerability is less severe in versions 6.0.0 to 6.1.4 due to prior security improvements but remains a concern if default security settings are disabled. Administrators are urged to upgrade to the latest versions to mitigate risks, and Grafana Cloud instances remain unaffected. A detailed timeline of the discovery and patching process is provided, along with guidance for reporting security issues and subscribing to security updates.

Patrick O'Carroll, a developer at Grafana Labs based in Stockholm, focuses on UX/UI design and frontend development, contributing to the Grafana open-source project. His current projects include designing new drilldown links and migrating styling from scss to emotion, alongside restyling components to adhere to a 4px grid. Although Patrick is not active on Twitter, his GitHub handle is ijin08. In his personal life, Patrick enjoys cooking, miniature wargaming, and swimming, though these activities have taken a backseat since he became a father. He prefers coding in silence and aligns himself with the Game of Thrones character Davos Seaworth. His recent TV binge was "The Last Kingdom," and he humorously remarks on the costumes of the Danes. Patrick prefers using spaces over tabs in his coding practices.

At GrafanaCon 2019 in Los Angeles, Peter Zaitsev, CEO of Percona, demonstrated how businesses can leverage Grafana and MySQL to create meaningful data visualizations that highlight important business metrics, such as sales per minute and revenue over 24 hours, utilizing existing client database information. Zaitsev emphasized the importance of visualizing data that is directly relevant to business goals, moving beyond standard performance metrics and addressing what truly impacts business outcomes. He also provided practical advice for optimizing MySQL setups, recommending the use of read-only users for safety, setting the max_execution_time variable to prevent long-running queries from overloading servers, and utilizing MySQL replicas for handling complex queries, ensuring efficient database performance during data visualization tasks.

The Trade Desk transitioned from a custom-built monitoring system to Prometheus, aiming to handle their extensive online advertising platform that processes 11 million requests per second. During a presentation at GrafanaCon, SRE Patrick O'Brien shared insights into the migration process, highlighting challenges such as migrating complex alerts, the need for improved Prometheus documentation, and addressing cardinality issues due to a high number of hosts. O'Brien emphasized the importance of creating a dedicated team, finding internal advocates to promote the new system, and actively participating in the community for better understanding and support. The transition also involved making metric names generic while embedding metadata to manage the overwhelming metrics data more effectively.

SevOne, a company specializing in network monitoring, has integrated Grafana into its operations to enhance the visualization of high-frequency telemetry data. Initially adopting Grafana for self-monitoring due to a lack of frontend engineers, SevOne discovered its potential for live streaming data sources, a feature not explicitly documented but achievable by wrapping data in an RxJS observable object. This approach allows them to bypass core changes in Grafana and manage extensive data points, catering to high-demand environments like IoT and emergency services. However, challenges such as infinite data loading, browser performance issues, and complex user experience persist, prompting the team to seek improvements. Despite these hurdles, SevOne remains committed to refining this functionality, with potential applications in various high-data scenarios.

KubeCon + CloudNativeCon EU in Barcelona will host 10,000 cloud native enthusiasts, featuring various events where Grafana Labs will have a significant presence. Key presentations include an introduction to Cortex by Grafana Labs’ Tom Wilkie and Weaveworks’ Bryan Boreham, a deep dive into Cortex, a keynote on the future of Observability with Tom and Red Hat’s Frederic Branczyk, and talks on Loki's progress and Kubernetes dashboards. The Grafana Labs Booth at the Sponsor Showcase will enable attendees to engage with the team and explore products like Grafana Enterprise and Grafana Cloud. Visitors can also learn about the latest updates in Prometheus with Goutham Veeramachaneni.

Hugo Häggmark, a fullstack developer focusing on frontend at Grafana Labs, is based in Vallentuna, near Stockholm, and heavily contributes to the Grafana open-source project, specifically working on enhancing the Explore feature. He left Twitter recently, but maintains a GitHub presence under the username hugohaggmark. Outside of work, Hugo enjoys family activities with his spouse and three children, alongside personal hobbies such as cooking, playing RPGs like Fallout, snowboarding, and free diving. His favorite tech gadget is the IoT kit from GrafanaCon, and he creates an ideal coding environment by using instrumental music for focus or enjoying ambient noise when less concentration is needed. Hugo identifies with the character Daenerys Targaryen from Game of Thrones and prefers using spaces over tabs in coding.

The ongoing conflict between open-source companies and public cloud vendors, exemplified by the disputes between Elastic N.V. and AWS, reflects a complex struggle over the monetization and ethical use of open-source software. Commercial open-source companies face criticism for imposing restrictions that contradict open-source principles, while cloud vendors are accused of profiting significantly from open-source projects without contributing back to the community. This dynamic has prompted companies like Grafana Labs to carefully strategize their approach, emphasizing partnerships with cloud vendors and maintaining a clear distinction between their open-source and commercial offerings. Grafana Labs aims to uphold community values while navigating the challenges of monetization, acknowledging the importance of transparency and community support for their business model. The situation underscores a broader tension within the tech industry over the balance between value creation and capture, with significant implications for the future of open-source ecosystems.

Bloomberg, primarily known for its media ventures, relies heavily on its Bloomberg Terminal, a platform delivering real-time financial data to over 325,000 global subscribers. To manage its vast data and ensure reliability, Bloomberg has focused on enhancing its telemetry systems. This effort, led by the central telemetry team, involves centralizing data collection, unifying alerts, and simplifying queries. By employing tools like Metrictank and Grafana, Bloomberg has streamlined its monitoring systems, encouraging collaboration among 5,500 engineers. The telemetry team has successfully centralized metrics, logs, and alerts, implemented speculative querying to optimize performance, and is working on improving dashboard discoverability to manage over 3,500 dashboards effectively. These initiatives aim to create a sustainable and reliable monitoring infrastructure, underscoring the importance of telemetry in maintaining Bloomberg's data services.

The tutorial on the Grafana Polystat Panel, specifically Part 2, expands on monitoring multiple Cassandra clusters within Kubernetes, utilizing a series of interconnected dashboards to provide an overview and detailed metrics. Starting with high-level rollups, the tutorial explains how to modify composites to represent the overall state of a cluster, integrating template variables for easier maintenance and customizing queries. It emphasizes creating a top-level view with the ability to drill down into more detailed dashboards, enhancing the visualization of metrics such as CPU usage. The guide highlights the potential for improvements like customizable tooltip widths, template variable usage in composites, and automatic composites to facilitate easier use of Polystat, while also encouraging users to share their dashboards on grafana.com.

Loki, developed by Grafana Labs, is designed to combine the simplicity of monolithic systems with the scalability of microservices, providing flexibility for users to start small and scale up as needed. Loki is a single binary with no dependencies, making it easy to deploy and manage, whether on a single laptop or across a large-scale microservices architecture. This approach allows users to incrementally grow their system by adding more components without the complexity typically associated with microservices, such as configuration management and troubleshooting. Inspired by projects like Thanos, Loki enables each instance to perform multiple roles or focus on specific tasks, offering flexibility in deployment and scaling strategies. This design caters to both newcomers who need an easy entry point and advanced users who require enterprise-level scalability and isolation of services.

Grafana is introducing a new visual component called the Bar Gauge as part of its updated panel and component architecture, which aims to enhance data visualization capabilities. Unlike traditional circular gauges, the Bar Gauge offers a more space-efficient and visually distinct way to present data by using a straight bar format that can be stacked horizontally. This visualization is versatile, allowing users to define thresholds with varying colors or gradients, mimicking retro LED displays to provide an intuitive sense of data status. The update also includes a new threshold editor and CSS animations to create smoother transitions on auto-updating dashboards. In addition to the Bar Gauge, Grafana plans to release further updates to the Singlestat panel and Table panel, alongside introducing multi-graph visualization and improved support for non-timeseries data. The Bar Gauge will be available in beta in the upcoming Grafana v6.2 release, and users are encouraged to try it out and provide feedback through GitHub.

Goutham Veeramachaneni discusses the process of automating the building of Grafana images on DigitalOcean using Packer, a tool he had previously used for AMI building and deployment. He highlights the manual process of setting up Grafana images, which involves installing and configuring the application, followed by cleaning the droplet of logs and keys. To automate this, Packer is utilized with its builders and provisioners, allowing the creation of images on platforms like AWS EC2 and DigitalOcean. Veeramachaneni details the challenges faced, particularly with updating the apt package database and resolving security update issues, which were eventually overcome by setting the DEBIAN_FRONTEND environment variable to noninteractive. Although the automation simplifies updating Grafana on the marketplace, it still requires manual execution on his laptop, with hopes to integrate it into Grafana's release process using CircleCI in the future.

Metrictank is developing a new feature called meta tags, sponsored by Bloomberg, to enhance the scalability and efficiency of their large-scale Graphite solution. This feature aims to address the challenge of adding numerous redundant tags to metrics without overwhelming system resources, by storing tags separately and associating them with metrics in a way that appears seamless to users. Unlike Prometheus, which requires explicit series joining for additional tags, Metrictank's meta tags offer a more transparent solution, allowing for native-feeling tags that facilitate filtering, grouping, and auto-completion without the need for explicit joins. The implementation involves using postings lists to link meta tag key/values to "metarecords," with an enrichment step to ensure the user experience remains smooth by adding meta tags as regular tags to returned series. The system is API-based, supporting operations like adding, removing, and managing tags across cluster nodes, and employs an eventually-consistent model for propagating tag rules. This development is seen as a significant advancement in providing a scalable and cost-effective way to enrich metrics with tags in Metrictank, aligning with customer demand and enhancing data querying capabilities in Grafana/Metrictank.

eBay's transition from custom UIs to Grafana plugins marked a significant shift in their monitoring and logging approach, as highlighted by Vijay Samuel at GrafanaCon. Initially, the quality of custom UIs for logging and metrics at eBay depended heavily on individual developers, leading to inconsistent experiences. Samuel's initial proof of concept with Grafana, although rudimentary, laid the groundwork for a more structured solution. The involvement of the Database Ops team, along with the adoption by eBay's SREs, facilitated the development of a dedicated data source plugin with enhanced features such as Docker support and Kubernetes deployment scripts. This shift allowed eBay to overlay data on graphs, catch issues like DNS flips, and provide hosted solutions, eventually leading to Grafana becoming a first-class citizen in their offerings. The monitoring team migrated to more cloud-native solutions for metrics and logging, while also contributing to open-source projects like Elastic Beats. This transformation eased the process of creating dashboards and demonstrated the power of Grafana to streamline monitoring efforts across the company.

At GrafanaCon 2019, Grafana Labs unveiled its latest cloud data source plugins, emphasizing its commitment to flexibility and avoiding vendor lock-in. The recent integration of Azure Monitor and Google Stackdriver plugins into Grafana v6.0 showcases this vision, offering users enhanced capabilities for visualizing and analyzing data across hybrid and multicloud environments. Azure Monitor's integration promises robust support, allowing seamless data visualization and alerting from various Azure services. Meanwhile, Google Stackdriver's collaboration with Grafana aims to provide users with advanced UI visualization options, SRE best practices, and in-context analysis tools, further enhancing the platform's open-source benefits. Oracle also joined the lineup with its plugin for Oracle Cloud Infrastructure, designed to support container-native workloads and surface metrics from OCI's monitoring service. These developments underline Grafana's ongoing efforts to democratize metrics and provide versatile, user-friendly solutions for a diverse range of cloud environments.

Grafana v6.1, released shortly after the major Grafana 6.0 update, introduces several enhancements aimed at improving the user experience and security. Key features include improved permissions allowing editors to own and administer dashboards, folders, and teams they create, fostering a more self-organizing team environment. The update also adds ad hoc filtering for Prometheus, enhancing data exploration by enabling the creation of key/value filters with autocomplete, applicable to all queries on a dashboard. Additionally, the release introduces the capability to list and revoke user authentication tokens via the API, setting the stage for more comprehensive security features. Minor updates include new keyboard shortcuts, bug fixes for Elasticsearch, and enhancements for datasource plugins, reflecting Grafana’s commitment to continuous improvement based on community feedback and contributions.

The Grafana Polystat Panel plugin is designed to consolidate multiple metrics and provide drilldowns to other dashboards, focusing on creating a panel for Cassandra using data from Prometheus collected from Kubernetes clusters. The tutorial explores creating a panel that displays CPU, memory, and disk utilization metrics, allowing users to group these metrics into an overall health status. It covers how to use simple PromQL queries, set thresholds for metrics, create composites to display combined metrics, and implement clickthroughs to navigate to detailed dashboards. The guide also discusses enhancing dashboard flexibility with Grafana template variables and concludes with a preview of more advanced features in the upcoming Part 2.

Prometheus 2.7 introduced subqueries, a feature that allows for more flexible ad hoc querying by permitting range vectors of function outputs, which was previously not possible due to concerns about performance and incorrect usage practices. Grafana Labs utilizes subqueries to streamline processes in billing and capacity planning. For billing, subqueries enable the calculation of P95 data points per minute without relying on recording rules, thus allowing more dynamic and experimental queries. Similarly, in capacity planning, subqueries facilitate the assessment of per-container CPU usage by combining multiple queries into one, enhancing efficiency in monitoring and resource allocation across Kubernetes clusters. Despite their advantages, subqueries require careful handling, particularly avoiding summation before applying the rate function to ensure accurate calculations.