July 2023 Summaries

Co-authored by experts from Amazon Timestream and Grafana Labs, the blog post discusses how integrating Amazon Timestream, a scalable and secure time series database, with Grafana's query caching can enhance the performance and cost-effectiveness of operational dashboards crucial for applications in DevOps, Security, and IoT. By leveraging Grafana's caching feature, which stores frequently accessed data, users can experience significantly reduced dashboard load times and lower query costs. This caching mechanism checks for data locally before querying the database, offering near-instantaneous reloads for similar time ranges. The post also explains how to set up Amazon Timestream with Grafana, including enabling cache settings, and highlights potential considerations like cache stampedes. Suggestions for overcoming these limitations and plans for future updates to Grafana are mentioned, alongside resources for getting started with both Amazon Timestream and Grafana.

The blog post, co-authored by LaunchDarkly Product Manager Tim Cook and Grafana Labs Director Logan Smith, discusses the integration of LaunchDarkly, a feature management platform, with Grafana, a popular observability tool, to enhance the monitoring and management of feature flags. This integration allows developers to gain real-time visibility into feature flag performance and usage, correlating this data with other metrics to make informed decisions. It streamlines collaboration by enabling the sharing of insights and dashboards within organizations and facilitates troubleshooting by identifying anomalies or regressions in feature releases. To utilize this integration, users must install the Grafana integration within LaunchDarkly and configure it to send annotation events to Grafana, allowing the creation of customized dashboards that incorporate feature flag data alongside other observability metrics. This collaboration aims to improve feature releases, optimize user experiences, and drive continuous improvement by leveraging data-driven insights.

Andreas Gerstmayr, a software engineer at Red Hat, discusses the Tempo operator, a tool designed to simplify the deployment and management of Grafana Tempo instances within Kubernetes clusters. Tempo, a distributed tracing system, uses object storage for trace data, making it compatible with various environments and offering features like TraceQL for advanced trace searching. The Tempo operator automates the creation and management of necessary Kubernetes resources, allowing system administrators to efficiently handle configuration settings and deployment options. By leveraging the Operator Lifecycle Manager or applying Kubernetes manifests, users can install the operator, which dynamically responds to changes such as load increases by adjusting resources like replicas. Additional functionalities include support for resource limits, multitenancy, Jaeger UI deployment, metrics exposure, and mTLS security, all aimed at easing operational tasks for administrators and site reliability engineers. The operator is continually being improved to introduce more self-healing functionalities, thereby enhancing the management of Tempo clusters in production environments.

Grafana 10 introduces a new user interface designed to simplify the configuration of SAML authentication, available for both Grafana Enterprise and Grafana Cloud's various offerings. This UI allows users to manage SAML settings without needing to restart Grafana or file support requests, offering real-time configuration validation and role-based access control (RBAC) to limit user access during setup. While both the configuration file and the new UI offer the same setup options, the UI streamlines the process with immediate feedback and easy modifications. Grafana advises against splitting SAML configurations across multiple sources due to potential confusion, suggesting a precedence order prioritizing UI settings, followed by environment variables, and then the configuration file. This update aims to enhance user experience and simplify authentication setup, with plans for additional interfaces to further ease the management of authentication and authorization in Grafana.

Grafana Cloud's Adaptive Metrics feature addresses the challenge of reducing observability costs by optimizing metrics management and cardinality, as many metrics generated are unused yet still incur costs. By aggregating unused and partially used metrics, Adaptive Metrics ensures users pay only for the metrics they use, leading to significant cost savings. Since its introduction, early adopters have reported a 25% reduction in time series, saving tens of thousands of dollars. The recent addition of a user-friendly interface enhances this feature by providing visibility and transparency in creating aggregation rules, allowing users to view, visualize, and download customized recommendations directly from the Grafana UI. Available across all Grafana Cloud tiers, the feature simplifies the process of applying recommendations through CLI or API, with significant metrics reductions visible in the billing dashboard. Grafana Labs is continuously enhancing Adaptive Metrics, with features such as automatic recommendations, robust search capabilities, and default availability in the Cloud Free tier, aiming to further streamline and maximize the utility of this cost-saving tool.

Interest in generative AI and large language models (LLMs) has surged, with tools like ChatGPT demonstrating the practical use of these models for generating prose, rewriting emails, and even coding. Grafana Labs has been exploring how to integrate LLMs into their open-source projects, particularly by enhancing their query editor with an LLM, starting with the Azure Data Explorer (ADX) data source. By employing a "Copilot model," similar to GitHub's Copilot, Grafana allows users to generate query suggestions that they can easily modify or discard, ensuring the LLM's potential inaccuracies do not disrupt workflows. The integration involves using OpenAI to convert natural language prompts into Kusto Query Language (KQL) snippets, with users providing their own OpenAI API keys to maintain flexibility and security. The initiative has been successful, prompting plans for further expansion and feature development, including potential enhancements to improve user experience and support for additional data sources.

A new Apache Mesos integration with Grafana Cloud offers an efficient way to monitor clusters by centralizing metrics and logs, leveraging the open-source Prometheus Mesos Exporter. This solution simplifies the management of cluster resources such as compute units, memory, and disk, and provides a prebuilt dashboard with over 20 metrics along with five alerting rules to track cluster health, resource usage, and system states. Users can set up the monitoring system by installing the necessary exporter on each cluster node and configuring the Grafana Agent. Once deployed, users gain access to a comprehensive overview dashboard and alerts for issues like high memory and disk usage, unreachable tasks, lack of cluster coordination, and inactive agents, all of which can be customized to fit specific needs. This integration aligns with Grafana Labs' goal of offering a consistent monitoring experience across various data sources, supported by a free-tier option for users to easily start tracking their clusters' performance.

Worldline, a leading payment service provider in Europe, utilizes Grafana Enterprise and Grafana Mimir to enhance its platform-as-a-service, accommodating its global operations and ensuring agile and faster transactions. The company's observability stack, which includes Grafana for visualization, Loki for logs, Tempo for traces, and Mimir for metrics, allows Worldline's internal teams, such as R&D and DevOps, to deploy assets and monitor infrastructure efficiently. The flexibility and scalability of Grafana, especially its multi-tenancy approach, enable Worldline to separate business lines securely and manage infrastructure as code. This observability as a service model supports Worldline's expansion into emerging markets by maintaining a standardized infrastructure across its data centers. The transition from Thanos to Grafana Mimir further addresses the need for scalable monitoring, while the potential adoption of Loki is expected to improve long-term logging capabilities. Worldline values Grafana's adaptability and continuous evolution, which aligns with its agile approach and commitment to staying competitive in the payment processing industry.

Grafana Labs, deeply involved in the OpenTelemetry community, has been enhancing the integration between OpenTelemetry and Prometheus, emphasizing interoperability and instrumentation SDKs. This guide highlights best practices for using OpenTelemetry to emit metrics and Prometheus for storage, addressing challenges such as metric normalization, resource attribute management, and naming conventions. Key recommendations include using full metric normalization, copying common resource attributes into metric labels, adhering to semantic conventions, and using seconds as the time unit for compatibility. It advocates for employing the Prometheus Remote Write exporter for efficiency and configuring out-of-order writes to manage data arrival discrepancies. Grafana Labs continues to lead in improving these open-source projects, focusing on further integration and introducing new functionalities like continuous profiling to OpenTelemetry.

Grafana Cloud has announced the general availability of its Frontend Observability service, which offers real user monitoring (RUM) to enhance the understanding of web application user experiences. This service, available to all Grafana Cloud users including those on the free tier, provides the ability to monitor frontend health, identify and resolve errors, and gain end-to-end visibility into user sessions. The tool measures Web Vitals like Time to First Byte (TTFB) and Largest Contentful Paint (LCP) to provide insights into user interactions, allowing for detailed analysis by dimensions such as device type and session ID. Frontend Observability in Grafana Cloud features automatic error grouping and ranking, facilitating quick troubleshooting by narrowing down the source of errors with JavaScript stack traces. The service integrates with Grafana Cloud Logs for querying and visualizing telemetry data, enabling correlation of frontend performance with business metrics. This holistic approach aids in identifying root causes by linking frontend user sessions with backend request traces, as exemplified by a case study involving a memory leak in a recommendation service.

Grafana Agent v0.35 introduces significant updates, including enhancements in horizontal scalability and ease of use with the new Flow mode, which offers a flexible, component-based version of Grafana Agent. The release simplifies migration by allowing conversion of Prometheus or Prometheus Agent configurations into Flow configurations, while also providing an option to run Prometheus configurations without committing to Flow mode. A notable feature is the beta clustering capability, which allows for auto-scaling of agent deployments using Kubernetes HorizontalPodAutoscalers based on memory or CPU load, and the ability to auto-distribute Prometheus and Pyroscope scrape targets. Additionally, new Prometheus exporter components have been integrated to facilitate monitoring of diverse production environments, and collaboration with Percona has enabled the development of a MongoDB exporter. Users are encouraged to explore the detailed documentation and participate in the Grafana Labs Community for further engagement and feedback.

Grafana Mimir, an open-source, scalable time series database, removed the use of mmap in its store-gateway component to enhance performance and stability. The store-gateway, responsible for fetching metrics data efficiently, previously used mmap to load index-headers, a technique that led to significant performance issues, including increased latency and health check timeouts due to Golang's inability to handle mmap's page faults effectively. The transition away from mmap involved adopting traditional file I/O operations and implementing several optimizations, such as buffered I/O, pooling file handles, and reducing unnecessary string creation, which collectively improved memory and CPU usage. These changes have virtually eliminated health check timeouts in Grafana Cloud Metrics clusters and are now available in the open-source version of Mimir as of v2.7. The development team plans to continue optimizing Mimir's components, including enhancing data streaming from object storage to the querier, with additional improvements to be discussed in future updates.

Celebrating its 10th anniversary, Grafana has introduced a range of features that enhance data visualization and ease of use for its users. Key highlights include Grafana Live for streaming data without external sources, query builders to aid new users, a command palette for efficient navigation, and role-based access control updates available in enterprise versions. Public dashboards allow sharing with non-Grafana users, while visualization suggestions offer intelligent options based on data context. Other notable features are the ability to convert data exploration sessions into dashboards, the use of chained variables for refined data filtering, the Infinity data source plugin for diverse API connections, and a plugin catalog for easy installation. Grafana 10 is designed to help users manage and share data insights effectively, with options to explore these features through community forums and events like GrafanaCON 2023. Users can experience the updated features through Grafana Cloud, which offers a free tier and various plans.

Kubernetes network and security monitoring can be enhanced using Hubble and Grafana, as discussed by Anna Kapuścińska, a software engineer at Isovalent. Hubble, built on Cilium, leverages eBPF technology to provide detailed observability of network events, making it easier to diagnose complex networking and security issues in Kubernetes environments. The integration with Grafana, through the Hubble data source plugin, enables visualization of network and security events, helping platform and application teams gain insights using features like flows query, process ancestry tree, and HTTP service map. These features allow users to analyze network flows, process events, and HTTP traffic within a Kubernetes cluster, providing critical insights for debugging and security assessments. The plugin also integrates with data stores like Hubble Timescape, Prometheus, and Grafana Tempo, making it a powerful tool for comprehensive monitoring. The Hubble data source plugin is in beta, with ongoing development to enhance its capabilities, and Isovalent invites user feedback through their community Slack channel.

Kubernetes Monitoring in Grafana Cloud has introduced updates that facilitate comprehensive oversight of resource utilization across past, present, and future timelines for Kubernetes users. This multi-purpose solution provides real-time insights into resource utilization, enabling teams to identify inefficiencies, bottlenecks, and potential outages, thereby optimizing performance and reducing costs. Historical data analysis helps establish benchmarks for improvement, while predictive intelligence allows for accurate forecasting of resource needs, aiding in better planning and budgeting. The platform's visualization capabilities offer both a broad overview and detailed insights into resource usage, empowering teams to make informed decisions about their infrastructure. Additionally, the integration of Grafana's k6 load testing tool provides further insights into the performance under varied loads, aiding in efficient resource allocation and minimizing risks of resource contention. Overall, these features contribute to more reliable infrastructure, enhanced application performance, and cost efficiency for organizations using Kubernetes.

Grafana, a data visualization platform, addresses the persistent issue of cross-site scripting (XSS) vulnerabilities, particularly DOM XSS, by implementing Trusted Types, an experimental JavaScript API developed by Google. This approach, combined with a strict Content Security Policy (CSP), aims to mitigate security risks associated with user-supplied data and third-party plugins, which often allow arbitrary HTML. Trusted Types enforce that only "trusted type" objects, not raw strings, can be passed to specific DOM APIs, thus preventing potential XSS attacks. Grafana's implementation includes creating policies for proper sanitization and using DOMPurify to support Trusted Types, ensuring that code is sanitized before execution. While challenges such as TypeScript support and plugin compatibility persist, Grafana's use of Trusted Types demonstrates a significant step towards enhancing security against DOM XSS, though it acknowledges that these measures cannot entirely eliminate the risk due to possible bypasses and limited browser support.

Grafana has revamped its alert state history system to enhance the analysis and tracking of alerting data. Previously reliant on an SQL-based system using the Annotations API, the new system now integrates Grafana Loki, a log database, allowing for more granular and efficient querying of alert state transitions. This change addresses previous limitations related to performance issues and inadequate retention schemes, particularly with high cardinality alerts. The new system includes a redesigned user interface that visually represents changes over time and provides detailed logs, improving the ability to observe trends and patterns in alert behavior. Users can now utilize multiple storage backends, including Loki, for better flexibility and transition ease. These enhancements aim to offer deeper insights into alert patterns, bolster system health, and reduce false positives, ultimately improving confidence in observability systems.

Relational databases like MySQL, PostgreSQL, and Oracle store valuable time series data that can be visualized using Grafana, an open-source analytics platform. This article demonstrates how to visualize time series data from SQL databases in Grafana, specifically using payment data from a MySQL database as an example. The tutorial explains how to use SQL macros in Grafana to create a time series panel, detailing the process of aggregating data into hourly blocks and employing macros to manage time ranges and data ordering. It emphasizes the benefits of macros, which simplify query modifications and ensure the dashboard reflects the correct time range. Additionally, the article addresses common errors, such as query syntax mistakes and excessive data returns, and offers troubleshooting tips using Grafana's Query Inspector. By applying these techniques, users can adapt the visualization process to various SQL databases and create effective dashboards. Grafana's flexible architecture allows for diverse time series visualizations, and its cloud service offers a free tier with access to Enterprise plugins, making it an accessible tool for managing metrics, logs, and dashboards.

Grafana Mimir, an open-source distributed time series database, has addressed a significant challenge of out-of-memory errors during complex queries by introducing innovative techniques within its store-gateway. Historically, Mimir struggled with unbounded memory utilization when processing high cardinality or long-time range queries, which could lead to system crashes. To mitigate this, the system now employs query sharding, allowing for concurrent execution across multiple machines, thereby reducing CPU and memory usage. Furthermore, the store-gateway has transitioned to a streaming model, which batches and progressively sends data to the querier, decoupling memory usage from data size and significantly reducing out-of-memory errors by 90%. This new approach not only maintains efficient memory utilization but also manages to lower latency through a preloading mechanism. Despite these advancements, challenges like managing a high number of inflight requests and optimizing series ID selection remain. Future improvements will continue to enhance the system's efficiency and scalability, as detailed in upcoming posts about further optimizations and enhancements in Grafana Mimir.

Dominik Süß provides a comprehensive guide on using the Grafana LGTM Stack in conjunction with Grafana Cloud k6 to identify and resolve performance bottlenecks. The article walks through setting up a load test on a well-instrumented web application, detailing how to simulate user flows with k6 scripts and analyze results in Grafana to track performance issues. By leveraging tools like Loki for logs, Tempo for traces, Mimir for metrics, and Grafana for visualization, users can pinpoint issues such as memory shortages and slow services. Süß suggests scaling services to distribute requests more efficiently and proposes further improvements such as caching and handling concurrent requests. He emphasizes the importance of regular load testing, integrating it into CI/CD pipelines, and exploring advanced observability features like exemplars and real user monitoring to enhance system reliability and performance.

Ed Welch's post discusses optimizing metric queries in Grafana Loki to enhance performance and accuracy, with a significant focus on using appropriate query syntax. For versions of Grafana older than 10.2.0, the post recommends using [$__interval] for range queries and [$__range] for instant queries to align with user expectations and prevent issues with data sampling. These practices help ensure that Loki queries data accurately, reflecting the actual count of logs over specified time ranges. The article explains the complexities of handling "step" and "range" values in queries, highlighting how Grafana's automatic interval calculation can sometimes lead to under- or over-querying, impacting the results. However, with the introduction of the [$__auto] feature in Grafana 10.2.0, users can simplify their queries, as this new variable automatically determines the correct settings for both range and instant queries, thereby enhancing usability and reducing the need for manual adjustments. The post hints at future developments that might further streamline query syntax, making it optional to specify ranges explicitly.