October 2024 Summaries

Grafana has significantly upgraded its dashboard architecture by migrating to the Grafana Scenes library, which provides a more stable, dynamic, and flexible backend while maintaining the same user interface. This change was necessitated by the limitations of their decade-old codebase, which hindered the implementation of user-requested features such as multiple time ranges and nested variable scopes. The new architecture, built on the Scenes library, offers an intuitive API for developers and aims to simplify complexity by managing dashboards as a tree structure, improving internal workflows and allowing for more efficient data handling. The migration process, which prioritized feature parity to avoid disrupting the user experience, has resulted in faster PDF exports, fixed bugs, and minor UI improvements like a fixed-position time picker and new Edit mode, while also laying the groundwork for future enhancements such as dashboard tabs and interactive metrics comparison. The updated architecture has already been rolled out to the majority of Grafana Cloud users, with plans for further features and improvements in the near future.

Grafana is a dynamic visualization tool that utilizes variables to enhance dashboard interactivity by allowing users to filter and focus on specific data points without having to edit queries or rebuild dashboards. Variables can be defined as dropdown lists, text boxes, or constant values, each serving different purposes such as filtering server data or switching between data sources. Grafana supports various types of variables including query variables, which populate dropdowns from data sources, and custom variables, where values are set manually. Additionally, interval variables allow users to compare data over different time periods, while ad hoc filter variables facilitate automatic filtering for certain data sources. Chained variables enable the use of one variable's value in another's query, and data links allow users to click directly on a dashboard element to filter data, streamlining the process of data exploration. This flexibility makes Grafana a powerful tool for creating reusable dashboards that can adapt to various data requirements and user preferences.

Grafana, a leading dashboarding tool, can be managed similarly to code by utilizing version control systems like Git, enabling collaboration and automated deployments. To facilitate editing and reviewing dashboards, tools such as Grafonnet and the Grafana Foundation SDK allow users to create dashboards as JSON files, which can be edited locally using integrated development environments (IDEs). Grizzly, a command-line tool, offers a serve function that allows users to validate and edit these dashboards in a Grafana instance before they are published. This process can be enhanced by ensuring that data source UIDs are consistent across development and production instances. Grizzly supports various programming languages, including Jsonnet, TypeScript, Golang, Python, Java, and PHP, allowing users to generate dashboards programmatically. By using Grizzly's watch feature, developers can monitor for changes in their dashboard files, facilitating immediate updates and previews in a Grafana instance, thereby streamlining the development and review process.

The GitHub data source plugin for Grafana, introduced in 2020, enables organizations to effectively visualize and analyze their GitHub activity through Grafana dashboards, offering insights into metrics such as daily GitHub use, reported issues, and more. Designed to address the limitations of GitHub's web interface, the plugin utilizes the GitHub API to fetch and display data, serving users across various roles like SREs, DevOps engineers, and product managers. It supports multiple query types including commits, issues, contributors, and pull requests, and introduces features like caching to improve performance and GitHub App authentication for enhanced security. Users can leverage variables and macros for dynamic queries and annotations to provide context in visualizations. The plugin is available on Grafana Cloud, Grafana Enterprise, and Grafana OSS, and is maintained by Grafana Labs. Comprehensive documentation and pre-configured dashboards are provided to facilitate ease of use, with ongoing feedback encouraged to enhance future iterations of the plugin.

ObservabilityCON 2024 offers a comprehensive suite of on-demand sessions for those unable to attend the event in New York, providing insights into the latest developments in Grafana Cloud and the Grafana LGTM Stack. The sessions feature keynote addresses from Grafana Labs executives and industry leaders from BlackRock, NVIDIA, and others, highlighting successful observability practices and the integration of AI/ML functionalities. They cover a range of topics, including cost efficiency strategies, unified observability, and the implementation of tools like Grafana Alloy and Beyla. Attendees can explore practical demonstrations and discussions on optimizing observability at scale, enhancing developer velocity, and utilizing new features for metrics, logs, and traces. Additionally, the sessions delve into user-centered observability approaches, leveraging synthetics, load testing, and real user monitoring. For those interested in more in-person learning, ObservabilityCON on the Road provides condensed events with technical deep dives and interactive sessions in various cities.

Grafana Cloud k6 has expanded its capabilities to include browser testing, allowing users to validate frontend web performance alongside backend API testing, which it historically supported. This new feature, powered by the k6 browser module, enables users to simulate real user interactions and collect frontend metrics to improve site reliability and performance. Browser tests in Grafana Cloud k6 can be written in JavaScript or TypeScript, similar to Playwright APIs, facilitating ease of use for those familiar with that platform. The testing framework provides comprehensive insights through performance overviews, cloud insights, browser timelines, and screenshots, offering a holistic view of a website's health. It supports various testing approaches, including hybrid and load testing, to ensure both backend and frontend components function optimally. Additionally, browser testing integrates with Grafana Cloud Synthetic Monitoring, enabling continuous validation of user journeys. Grafana Cloud k6 is positioned as a comprehensive platform for ensuring web applications meet user expectations, with a free tier and scalable plans available.

Grafana 11.3 introduces significant updates, including the general availability of Scenes-powered dashboards, which offer a more stable, dynamic, and flexible architecture. This release enhances the dashboard experience by allowing API calls from canvas elements, introducing recording rules for Grafana-managed alerts, and offering Explore Logs for easier log visualization and troubleshooting without complex queries. Visualization improvements include auto-formatted table cell values and new action capabilities in various chart types. Additionally, role-based access control for notifications is available in Grafana Enterprise and Grafana Cloud, and new authentication features, like GitHub App authentication for GitHub data sources and LDAP configuration through the UI, are included. Users can also create correlations to external URLs for better data linking, and Grafana Alerting has a simplified query section for alert rule creation. The release emphasizes community engagement and encourages users to explore its new features via Grafana Cloud's free tier.

Grafana has introduced queryless telemetry exploration experiences called Explore apps, designed to accommodate users with varying levels of expertise in observability. These apps, which now include Metrics, Logs, Traces, and Profiles, offer no-code, low-code, and code options to help users derive insights from telemetry data without the need to learn complex query languages. This initiative aims to make observability tools more accessible and user-friendly by removing the barrier of mastering query languages, thus enabling users—from beginners to experts—to efficiently monitor and troubleshoot their systems. The Explore apps are part of Grafana's broader strategy to support a diverse user base, and they have been well-received since their launch, with plans for continued enhancements and user feedback integration. Grafana's commitment to evolving its LGTM Stack reflects its dedication to optimizing the observability experience for all users, offering both cloud and open-source solutions to cater to various deployment needs.

Generative AI, renowned for its ability to generate diverse content such as text, images, and music, presents challenges in monitoring and optimization, which traditional observability tools struggle to address. Grafana Labs offers a solution through its AI Observability feature in Grafana Cloud, leveraging OpenLIT, an open-source SDK designed for monitoring generative AI systems. This integration allows users to track model performance, optimize costs, and trace data flow comprehensively while offering insights into AI models' nuances via a unified interface. It benefits AI engineers, data scientists, and DevOps professionals by enhancing the reliability and efficiency of generative AI applications. With easy setup and auto-instrumentation for over 30 generative AI tools, including LLMs and various frameworks, the AI Observability feature integrates seamlessly with existing systems, adhering to industry observability standards. Users can monitor metrics such as request rates, costs, and latency, allowing for informed optimizations, and can easily access detailed traces and request information to refine their AI implementations.

Monitoring Kubernetes requires a novel approach beyond traditional techniques due to its complex nature and the intricacies involved in managing cloud-native architectures. In a podcast episode of "Grafana's Big Tent," experts from Grafana Labs, including Mat Ryer, Tom Wilkie, Vasil Kaftandzhiev, and Dio Tsoumas, discuss the challenges and responsibilities associated with Kubernetes observability. The discussion highlights the shift towards platform-based monitoring, where the infrastructure offers built-in observability capabilities, thereby alleviating individual engineers' burden. The episode also delves into the financial implications of monitoring, emphasizing the need for cost attribution and resource management to ensure efficiency. Kubernetes' integration with Prometheus and its rich metadata capabilities allow for enhanced monitoring and dashboarding, offering developers deeper insights into their applications' performance and facilitating efficient troubleshooting. The community's contribution to the development of dashboards and alerts has made DevOps tasks more accessible, though comprehensive observability still requires a combination of metrics, logs, and performance data. Overall, the episode underscores Kubernetes' distinctive monitoring needs and the evolving landscape of observability tools designed to support it.

Grafana Labs recently addressed a critical security vulnerability, CVE-2024-9264, affecting Grafana versions 11.0.x to 11.2.x, which allowed for command injection and local file inclusion through SQL Expressions, an experimental feature that was unintentionally enabled by default. This vulnerability, with a CVSS score of 9.9, could potentially allow users with Viewer permissions or higher to access any file on the host machine, including unencrypted passwords, provided the DuckDB binary was accessible in the system's PATH. The issue, discovered internally in September 2024, prompted Grafana to release patched versions, remove the SQL Expressions feature, and ensure the security of their cloud offerings. Users are advised to upgrade to the patched versions or remove the DuckDB binary to mitigate the risk. Grafana Labs also emphasizes responsible disclosure of security issues and provides a platform for reporting vulnerabilities.

The guide provides a detailed walkthrough on setting up Grafana Cloud Application Observability using the OpenTelemetry Operator within a Kubernetes cluster, emphasizing the importance of monitoring application health similar to personal health by tracking vital signs. It introduces the concept of zero-code auto-instrumentation, which alleviates the burden of manually configuring traces, thereby enhancing productivity for developers and SREs. The OpenTelemetry Operator simplifies the process by automating the injection of necessary instrumentation components, allowing for efficient root cause analysis in distributed systems. Key steps include deploying the OpenTelemetry Operator Helm chart, configuring Kubernetes Instrumentation CustomResources, and validating deployments using tools like K9S or kubectl. The guide also discusses troubleshooting common issues and the optional step of annotating Kubernetes namespaces for consistent application tracing. Grafana Cloud offers a user-friendly platform with a free tier for monitoring metrics, logs, and traces, encouraging users to explore auto-instrumentation further with tools like Grafana Beyla.

Grafana Cloud has introduced a range of updates to enhance its fully managed observability platform, including new features in Synthetic Monitoring powered by Grafana k6, such as k6 browser checks for frontend performance metrics and scripted checks for complex applications. Improvements have also been made to the data visualization experience, with a more intuitive dashboard-sharing process and the ability to add actions to various visualization types. Authentication and authorization have been streamlined, with easier setup for LDAP and SAML certificates, while a new escalation step in Grafana OnCall allows for automatic incident declaration. Additionally, updates to data source plugins now include managed plugins for Grafana Labs-created sources and GitHub App authentication for the GitHub data source, offering enhanced security and ease of use. These enhancements are available for free with Grafana Cloud's generous Cloud Free plan.

Grafana Labs is expanding its Enterprise data source plugins, allowing users to visualize data from a variety of external systems within Grafana dashboards, with recent additions including Atlassian Statuspage, Cloudflare, and Netlify, all currently in public preview. These integrations enable users to access detailed analytics and insights, such as service outages and maintenance windows from Atlassian Statuspage, DNS and internet traffic data from Cloudflare, and site, build, and deploy information from Netlify. Grafana's approach aligns with its open philosophy of offering users the ability to derive meaningful insights from their data regardless of its origin. Additionally, Grafana has introduced a public roadmap for plugin development, encouraging community engagement and feedback, and provides a free tier in Grafana Cloud for users to explore these Enterprise plugins.

Prometheus is portrayed as a comprehensive monitoring framework that extends far beyond being a simple time series database, offering robust capabilities such as metric scraping, querying, alerting, and dynamic service discovery. The book "Prometheus Up & Running" by Julien Pivotto and Brian Brazil is highlighted as an essential resource for Grafana users seeking to deepen their understanding of Prometheus's functionalities. It explains that Prometheus leverages mechanisms like client libraries, node exporters, and service discovery to efficiently collect metrics without overwhelming systems, while PromQL is emphasized as a powerful querying language essential for complex data handling and alerting. The text encourages Grafana users to utilize Prometheus's strengths, such as recording rules for optimizing data granularity and performance, to enhance their observability practices. The post concludes by recommending the book for its detailed insights and practical exercises, making it a valuable resource for mastering Prometheus.

Grafana recently undertook a significant project to decouple its Prometheus data source from the core Grafana application, aiming to create vendor-neutral libraries that allow for the development of custom Prometheus data sources. This initiative involved deprecating AWS authentication from the original Prometheus data source and creating a dedicated Amazon Managed Service for Prometheus plugin to cater specifically to AWS users, all while maintaining stability in the core data source. By focusing on modularity and reusability, Grafana Labs enabled external applications to utilize the Prometheus data source code and develop new data sources tailored to specific needs without being tied to the Grafana release cycle. This project required a comprehensive rewrite of the codebase, including the separation of frontend and backend code into GoLang workspaces, which facilitated better maintainability and scalability. The team enhanced their CI/CD processes and increased testing coverage to ensure seamless functionality, despite pausing new feature development during the transition. The outcome is a robust framework that adheres to good engineering practices, setting the stage for future growth and improvements in observability for third-party data sources, with plans to expand this approach to other vendors like Microsoft Azure.

Bryan Boreham, a Distinguished Engineer at Grafana Labs, explores the intricacies of PromQL, the query language for Prometheus, aiming to provide a comprehensive understanding of how it processes queries. Despite being a Prometheus maintainer, Boreham found many aspects of PromQL obscure, prompting him to delve into its mechanics and share his insights at PromCon 2024. PromQL is crucial for expressing which metrics to use and operations to perform, typically for dashboard visualization, with two primary query types: instant and range. The execution of a PromQL query involves parsing text into an Abstract Syntax Tree (AST), which the engine processes to pull data from storage, employing selectors, functions, aggregations, and operators to compute results. Boreham discusses the internal workings of PromQL, including how data is fetched, processed, and output, along with the tools and resources available for learning and optimizing queries. He emphasizes the role of PromQL functions, aggregations, and binary operators, and how they interact to perform complex data manipulations, while also highlighting the significance of tracing and profiling for gaining deeper insights into query execution.

Grafana, an observability platform renowned for its ease of use, offers a range of features designed to help users quickly build effective dashboards. Users can add data sources, such as Grafana Loki, to visualize data from diverse origins, and Grafana provides step-by-step guidance for setting up these sources, including the configuration of authentication details. When creating dashboards, selecting the appropriate visualization type is essential, with options like time series graphs, stat panels, and bar charts available to suit various data stories. Additionally, Grafana allows users to create alerts to monitor specific conditions, such as high service latency, by defining alert rules, contact points, and notification policies to ensure timely attention to critical issues. Annotations can be added to visualizations to indicate significant events, offering contextual insights and facilitating team collaboration. For further learning, Grafana provides a "Grafana for Beginners" video series on YouTube, along with extensive documentation to assist users in mastering these functionalities.

Adaptive Metrics, a feature in Grafana Cloud, addresses the challenge of metric overload by aggregating unused and partially used metrics into lower cardinality versions, helping to cut costs and reduce noisy data signals. Highlighted in the podcast "Grafana's Big Tent," the feature was developed to combat the common issue of explosive growth in time series data due to teams' tendencies to collect extensive metrics without considering monitoring costs. Grafana Labs Engineering Director Mat Ryer, alongside engineers Patrick Oyarzun and Mauro Stettler, discussed how Adaptive Metrics offers recommendations for reducing cardinality by identifying unnecessary labels and allows users to implement these recommendations through a metrics ingestion aggregator. The feature's dynamic capability to adapt over time and generate new recommendations based on usage patterns distinguishes it from traditional solutions that rely on static lists of essential metrics. Additionally, the discussion touched on the potential future of Adaptive Metrics and observability, where a "just-in-time metrics" approach could enable organizations to rapidly increase data collection during incidents for thorough investigation and analysis without incurring consistently high storage costs.

Grafana Labs has developed an anomaly detection framework using PromQL to enhance incident investigation by providing crucial context quickly, which is essential during time-sensitive scenarios like on-call alerts. The framework, designed to work seamlessly with Prometheus-compatible systems, operates without external dependencies and is scalable to handle large metric data volumes. Initially based on the z-score formula for anomaly detection, the framework uses Prometheus recording rules to establish baselines and detect anomalies by setting upper and lower behavior bands. Challenges such as extreme outliers, low sensitivity, and discontinuities were addressed by introducing smoothing functions, filtering low variability periods, and defining minimum margins. The framework also accounts for long-term recurrent patterns by predicting behavior based on past data. Users can implement the framework by adding specific recording and alerting rules to their Prometheus instance, and Grafana Labs encourages feedback for future enhancements. The framework's effectiveness is demonstrated within Grafana Cloud, where it integrates with SLO-based alerts to provide actionable insights and accelerate root-cause analysis.

Containers, which package software to run consistently across various environments, often use microservices architecture to enhance performance, flexibility, and scalability. Observability is crucial in these distributed systems to ensure smooth operations, and Grafana is a powerful tool for monitoring these environments by allowing users to query, alert, and explore metrics, logs, and traces stored across different platforms. Grafana provides customizable dashboards for visualizing metrics, which helps in identifying and resolving issues, optimizing performance, and making informed decisions. The documentation from Grafana Labs covers deployment and configuration using Docker, Kubernetes, and Helm Charts, offering resources for both Grafana Enterprise and Grafana OSS users. This documentation includes guides on creating customized setups, persistent storage options, and troubleshooting practices. Community contributions are encouraged, with options for providing feedback, reporting issues, and engaging in the Grafana Champions program to support broader participation.

ObservabilityCON 2024, held in New York City, highlighted the latest updates and features in Grafana Cloud and the Grafana LGTM Stack, offering technical sessions, user success stories, and community engagement. For those unable to attend, Grafana Labs is bringing ObservabilityCON on the Road to various cities, including Berlin, London, São Paulo, and Paris, with plans to expand to North America and Australia. These events will replicate the conference experience with technical deep dives, live demos, and expert consultations. Attendees will have the opportunity to learn about new tools like Grafana Alloy and Beyla, as well as best practices for enhancing observability strategies. Local experts will also share real-world insights, making these events an invaluable resource for advancing technical skills and connecting with the observability community.