August 2023 Summaries

Grafana Labs has significantly enhanced its Grafana OnCall service to improve the on-call experience for teams, focusing on reducing the stress and anxiety often associated with on-call duties. Key upgrades include the introduction of a web-based scheduling interface that allows easy schedule creation and editing, complete with features like time zone visualization and schedule quality scoring to prevent common scheduling issues. A newly launched native mobile app for iOS and Android lets users manage alerts directly from their devices, even overriding do-not-disturb settings for critical notifications, and offers customizable notification policies. Additionally, Grafana OnCall now supports email notifications, expanding its integration capabilities beyond SMS, Slack, and Teams, which is particularly beneficial for teams that rely heavily on email communication. These improvements aim to make on-call management more intuitive and less intrusive, aligning with Grafana OnCall's broader integration with Grafana Cloud's Incident Response Management offerings.

Grafana JSON API plugin allows users to import and visualize third-party data sources in Grafana Cloud by querying publicly available JSON endpoints, making it an excellent tool for exploratory data analysis in a no-code environment. Users can quickly create dashboards by pulling data from any URL that returns JSON, including REST APIs or static file servers. The guide provides a step-by-step tutorial on building a dashboard using the Star Wars API as an example, demonstrating how to query JSON data and visualize it using Grafana's tools. The tutorial illustrates the process of creating a data source, building a dashboard, adding visualizations, and executing queries with JSONPath, highlighting the ease of visualizing complex data sets without extensive technical expertise. Additionally, Grafana Cloud offers a free tier, making it accessible for users to experiment with and monitor their data.

Grafana Incident, part of the Grafana IRM suite in Grafana Cloud, now supports Microsoft Teams integration, enhancing its existing range of tools like Zoom, Google Meet, GitHub, and JIRA. This integration allows for seamless incident management by centralizing communications and updates within Microsoft Teams, thereby reducing context switching and improving response efficiency. It supports key operations such as comment updates, role assignments, incident severity adjustments, and task management directly in Teams, streamlining the incident timeline and post-incident review process. The Microsoft Teams integration is easily installable through Grafana's integrations page and enables real-time synchronization of incident information, ensuring that all updates are reflected across platforms, whether accessed through Teams or the web UI. The Grafana IRM chatbot provides a comprehensive suite of commands for managing incidents, and the integration aims to reduce the mean time to resolution (MTTR) by minimizing platform juggling and enhancing collaborative problem-solving during incidents.

Grafana Cloud offers enhanced capabilities for centralizing AWS observability by allowing users to connect over 60 popular AWS services, such as EC2, Lambda, and S3, into a single platform for streamlined monitoring and troubleshooting. This integration eliminates the need for dispersed observability tools and utilizes a fully managed backend with support for PromQL, resource-based tagging, and unlimited queries, promoting greater efficiency and data consistency. Users can easily configure AWS connections without sharing credentials, access prebuilt dashboards, and benefit from simplified billing through the AWS Marketplace. Grafana Labs plans to expand its AWS integrations further, adding support for more services, deployment types, and curated dashboard experiences to enhance user insights into AWS environments.

Grafana Pyroscope 1.0 represents the culmination of merging two open-source projects, Pyroscope and Grafana Phlare, into a unified continuous profiling tool designed to provide developers with an efficient and scalable platform for analyzing profiling data. This release builds on the strengths of both projects to offer a comprehensive solution with low overhead and robust server-side capabilities, capable of handling large-scale data ingestion and analysis. Grafana Pyroscope 1.0 is integrated into the Grafana Cloud observability stack, which also offers a managed service called Grafana Cloud Profiles, enabling users to easily correlate profiling data with logs, metrics, and traces. The development team aims to expand its features, including improved workflows, better integration with OpenTelemetry, enhanced client-side management, and eBPF support, while also providing training resources to facilitate user adoption. The platform is designed to seamlessly fit into existing workflows, optimizing infrastructure and improving application performance, with free and paid tiers available for users.

Grafana Labs is actively exploring the role of generative AI in enhancing its observability tools, particularly focusing on bridging the gap between users and the vast scale of data they handle. The company has introduced Grafana Incident auto-summary, a feature that leverages large language models (LLMs) to generate concise summaries of incident timelines, aimed at reducing the manual effort required in post-incident reviews. This feature, enabled by an OpenAI integration, is available across all tiers of Grafana Cloud, including the free tier. Grafana Labs emphasizes an open-source approach, integrating generative AI capabilities into its platform to allow users to easily add LLM-powered features to their own plugins. This initiative is part of a broader vision to enhance data query and visualization functionalities within Grafana, with ongoing efforts to develop tools and frameworks that the community can build upon. The company encourages community feedback and participation in this evolving AI-driven landscape, maintaining its commitment to transparency and open collaboration.

In response to the pressure on organizations to reduce costs, particularly in SRE and platform engineering teams, many have struggled with managing metrics without compromising observability. Grafana Cloud offers a solution by providing a comprehensive framework and features for cost-effective metrics management. The platform focuses on four key areas: inspecting cost drivers, attributing costs to specific teams or environments, optimizing telemetry by identifying and reducing unused metrics, and continuously monitoring to prevent unexpected cost spikes. Grafana Cloud's tools, such as Adaptive Metrics and cardinality management dashboards, help streamline the process of managing metrics, allowing organizations to scale and maintain predictable costs. This approach contrasts with traditional methods like creating data silos, building in-house solutions, or doing nothing, which often lead to inefficiencies and increased technical debt. By leveraging Grafana Cloud, organizations can gain transparency and control over metrics usage, ultimately enhancing their observability strategies without unnecessary financial strain.

Grafana 10.1 introduces several key features and improvements aimed at enhancing observability insights, including updates to flame graphs, a new geomap network layer, and improved alerting workflows. The revamped flame graph visualization now offers a detailed view of application performance, while the geomap network layer simplifies the visualization of network spatial distribution by allowing connections to be defined without multiple layers or transformations. The release also addresses data accuracy in dashboards by enabling users to disconnect values above a specific threshold to prevent misleading visualizations. Additionally, span filtering in trace analysis and TraceQL response streaming enhances the ability to pinpoint specific spans and view partial query results in real-time. Grafana has also introduced UI improvements and a new alert rule creation workflow, distinguishing between Grafana-managed and data source-managed alerts for more streamlined management. These updates, along with experimental features like the separation of visualization panels and widgets, aim to improve the overall user experience and dashboard creation process.

Grafana k6 v0.46.0 introduces significant updates, including the ability to configure TLS per gRPC connection, enhanced tracing support with k6 x Tempo integration, and a more efficient cloud output protocol using binary-based payloads over HTTP. The update also brings changes to the experimental k6 browser module, allowing automatic handling of browser lifecycles and requiring the specification of browser type in scenario options. Grafana Cloud k6 now offers features such as scheduled tests, activity panels, and customizable PDF reports, while the k6-operator pre-release candidate introduces the Private Load Zones API for cloud infrastructure testing. The extension ecosystem has been actively updated to meet new standards, with several new extensions added, and the k6 Champions program has welcomed new contributors, encouraging community engagement and advocacy.

Grafana Labs experienced a security incident involving the unintentional exposure of a GPG private key and passphrase, necessitating a GPG key rotation for users who install Grafana via package repositories such as apt and yum. The company has revoked the compromised key and issued a new one, urging affected users to update their trusted certificates to maintain security. Users who installed Grafana through other means like OS package repositories, Docker images, or Grafana Cloud are not impacted by this incident. Grafana Labs emphasizes transparency and has published a post-incident review, while continuing to provide detailed instructions for users to update their GPG trust relationships. The company also encourages reporting potential security vulnerabilities via a dedicated email and provides security announcements on their blog.

Grafana Cloud Logs faced significant challenges with its Grafana Loki logs database when traffic surpassed the limits of cloud vendors' object storage services, prompting a comprehensive redesign of its caching strategy. The service initially used memcached clusters sized for recent data, but diverging access patterns and excessive cache churn led to inefficiencies and costly object storage retrievals. To address these issues, the team transitioned to using local SSDs for caching, which dramatically increased capacity from 1.2TB to 50TB while reducing costs by 98%. This change, facilitated by memcached's extstore feature, significantly improved cache hit rates, reduced requests to object storage by 65%, and effectively eliminated rate-limiting issues. Despite increased latency due to the introduction of disks, the solution proved effective, showcasing a careful balance of trade-offs in software engineering, with a noteworthy reduction in operational costs and improved performance.

ObservabilityCON 2023, Grafana Labs' annual open-source observability conference, will be held in London on November 14-15, featuring a variety of activities including keynotes, technical workshops, and community success stories. The event will highlight the latest trends in observability and updates to the Grafana LGTM Stack, which includes components like Loki, Grafana, Tempo, and Mimir. Participants can engage in hands-on sessions to enhance their skills in creating dynamic Grafana dashboards, and they will have opportunities to network with other community members and experts. Registration opens in mid-September, with the full agenda available in early October, and both speaking and sponsorship opportunities are available for community members who wish to contribute.

Over the past six months, Grafana Mimir has undergone significant optimizations to enhance query performance and cost efficiency, particularly by reducing memory usage in the store-gateway component. This was achieved by eliminating inverted index lookups for broad matchers, which significantly reduced memory spikes that previously led to out-of-memory (OOM) errors. The optimization process involved continuous profiling and active queries tracking to identify and address memory consumption issues, notably those caused by unnecessarily loading large volumes of series IDs. By leveraging the structure of the Prometheus TSDB, Grafana Mimir was able to fetch only the necessary series and apply matchers more efficiently, resulting in a reduction of store-gateway memory utilization by up to 64% in some clusters, and a 30% reduction in memory requests across all Mimir clusters. The improvements not only stabilized query performance but also contributed to lowering the total cost of ownership (TCO) for Mimir. Future enhancements are anticipated to further refine the querying experience by adopting streaming techniques in the PromQL evaluation.

Grafana Labs' hiring process for backend engineers is designed to assess a candidate's potential performance and fit within the company through a series of structured interview stages. The process begins with an initial phone call with a tech recruiter, followed by four interview stages: the hiring manager interview, a coding exercise, a non-abstract system design exercise, and a reflection and offer stage. Each phase aims to evaluate not only technical skills but also problem-solving abilities, communication, and cultural fit. The coding exercise emphasizes collaborative problem-solving and communication, while the system design exercise focuses on a candidate's ability to conceptualize and reason about complex systems. Throughout the process, candidates meet various team members, providing a comprehensive view of their suitability for the role and offering them insight into the team and company culture. The final reflection stage allows both the candidate and Grafana Labs to assess mutual compatibility before making an employment offer.

Victor Sonck, a Developer Advocate for ClearML, discusses how to effectively monitor machine learning models in production using Grafana integrated with ClearML Serving, a module built on NVIDIA Triton Inference Server. ClearML, an open-source MLOps platform, aids in model deployment and management by offering tools for seamless integration into the MLOps stack. Grafana, along with Prometheus, is utilized to monitor model predictions and performance, enabling users to gain insights, detect model drift, and receive alerts for abnormal model behavior. ClearML Serving facilitates model deployment by connecting inference servers to model repositories, supporting canary deployments, and automating logging and deployment processes. It captures serving metrics and sends them through Apache Kafka to a custom statistics service, which processes and reports them to Prometheus, allowing Grafana to visualize and analyze the results. The integration supports the creation of custom alerts within Grafana, notifying users of issues such as latency or data drift, thus allowing immediate intervention to maintain model accuracy.

The Grafana Champions Program has been launched to acknowledge and celebrate the contributions of individuals who are instrumental in advancing the Grafana open-source community. Developed by the Grafana Developer Advocacy team, the program aims to recognize community members who actively contribute to the community by sharing knowledge, offering support, and promoting discussions around observability. The inaugural champions, including individuals like Grant Pinkos and Mikhail Volkov, have been recognized for their significant contributions through tutorials, community discussions, and open-source development. The program encourages participation from individuals who are passionate about mentoring and contributing across the Grafana LGTM Stack, offering benefits such as invitations to events and increased visibility. Applications are open to global contributors, with a focus on inclusivity and creativity in contributions, which can range from blog posts to conference presentations and code contributions.

DHL Express Switzerland has significantly enhanced its IT and business flows by implementing a Grafana-centric observability solution, as detailed in a GrafanaCON 2023 presentation by Head of IT Djamel Djedid and Lead Architect Michael Lerch. The initiative began in 2020 with a proof of concept using Grafana and Prometheus to address the limitations of their legacy monitoring system, enabling the team to proactively resolve issues and reduce manpower needs. By 2021, they expanded the implementation with Grafana Alerting and integrated it with Microsoft Teams, making observability a default attribute of new applications, which improved customer satisfaction during a business surge. In 2022, they used Grafana k6 for performance testing to ensure a smooth transition from on-premises data centers to the public cloud, confirming the cloud's stability for more users. By 2023, the observability stack included 80 alerts, 40 dashboards, and 60 active users, with plans to further integrate Grafana OnCall and Grafana Loki, providing a comprehensive and efficient system status overview.

Brian LeBlanc shares his experience of developing a remote pool monitoring system using a combination of IoT devices, Grafana Cloud, and other technologies to prevent his pool pump from running dry while he was away. After discovering costly retrofitting options, he opted for a cost-effective mechanical device called a Pool Sentry to maintain water levels, paired with a LinkTap G2S smart water controller to measure water consumption. By integrating these devices with a Raspberry Pi, Prometheus, and Grafana Cloud, he created a comprehensive dashboard to visualize and track metrics such as water usage, temperature, and rainfall. This setup, which included a Tempest weather station for additional environmental data, allowed him to ensure optimal pool conditions remotely during his vacation. LeBlanc's project not only provided peace of mind but also sparked further interest in utilizing Grafana for additional pool monitoring capabilities.

Qonto, a fintech company focused on simplifying banking processes for SMEs and freelancers, transitioned from using Elasticsearch to Grafana Loki for its network observability platform to manage massive amounts of network logs more cost-effectively. The decision came in response to the high storage costs associated with Elasticsearch, despite its capabilities, and the prohibitive pricing of AWS's native toolchain. Since implementing Loki in January 2023, Qonto has been able to handle 30 TB of compressed data monthly with improved efficiency in searching and filtering logs using Loki’s LogQL. The new architecture leverages AWS resources such as S3 buckets and Lambda functions for log ingestion, with Prometheus and AlertManager used for metrics and alerts. The switch to Loki has facilitated the detection of network traffic issues swiftly, making the task nearly enjoyable for the SRE team, and has prompted further expansion of its application to other AWS services, demonstrating the versatility and effectiveness of open-source solutions in enhancing network reliability and cost management.

Grafana k6 is a versatile load testing tool designed to simulate various traffic patterns and analyze software performance under different conditions, helping developers identify bottlenecks and ensure software reliability. It is structured around four main pillars: scripting workloads with JavaScript using the goja interpreter, planning and executing tests with configurable options, collecting and aggregating performance metrics, and forwarding results to external outputs like JSON, CSV, or platforms like Prometheus and InfluxDB. The tool functions as a simplified work scheduler, orchestrating execution plans and efficiently managing workloads by utilizing available hardware resources. This allows users to gain immediate insights from test summaries and perform in-depth analysis with tools like Grafana. With its extension and library system, k6 supports various open-source technologies, enabling customization to meet specific testing needs. Users can begin exploring k6's capabilities with the free tier of Grafana Cloud, which provides a range of resources for load testing and performance analysis.

Grafana Cloud Profiles, now generally available, integrates continuous profiling with traditional observability pillars—logs, metrics, and traces—offering a unified view of system performance. This approach challenges the conventional "pillar" metaphor of observability by highlighting how interconnected these components are, enabling more comprehensive insights into system health. Continuous profiling provides detailed analysis down to the code level, helping identify root causes of issues such as CPU spikes and memory inefficiencies, thus optimizing infrastructure, simplifying debugging, and enhancing performance. Companies like Alpaca and Zomato have leveraged this integration to maintain high performance and service quality, underscoring the benefits of a cohesive observability platform. Grafana Cloud Profiles can be integrated using the Grafana Agent or Pyroscope SDKs, with a generous free tier available for new users.

Grafana Labs has introduced several enhancements to its distributed trace visualization features, aiming to simplify the process of tracing for users and integrate it into a mature observability strategy. Key updates include an improved trace view header that highlights primary information more prominently, such as service and operation names, and uses color-coded tags for status codes. Additionally, span filtering has been introduced to allow users to search and highlight spans based on specific criteria, making it easier to identify performance issues or errors. A new "Show matches only" feature further aids in reducing cognitive load by displaying only the spans that match selected filters. Improvements have also been made to the configuration editor for tracing data sources like Tempo, Jaeger, and Zipkin, organizing settings into collapsible sections and providing more context and documentation links. The updates, available in Grafana 10.1, are complemented by enhanced documentation and a feedback mechanism to capture user insights, with the goal of making tracing more accessible and efficient for organizations.

Adaptive Metrics, a new feature introduced by Grafana Cloud, addresses the challenge of managing the high cardinality of Prometheus metrics to optimize data value and reduce costs. Highlighted in a recent webinar, this feature allows users to significantly decrease their metrics' footprint and save on observability expenses by recommending and applying metric aggregations. The feature has been tested to scale efficiently with millions of metrics, and though automation of these recommendations is currently user-driven, future developments may include built-in automation. While the recommendations are not based on machine learning, they are derived from recent data usage patterns, ensuring no disruption to existing dashboards or queries. Adaptive Metrics is available across all Grafana Cloud tiers, including the free version, and does not yet support on-premises databases. Although the feature currently lacks options to bookmark or control the sensitivity of recommendations, it provides a flexible approach to metric management that can adapt to evolving data access patterns while maintaining access to essential metrics.

Christopher Field and Mikhail Volkov detail their journey to creating an innovative Grafana dashboard, which has become a finalist in the Grafana Golden Grot Awards for its significant impact on scientific and engineering experiments. The dashboard, developed through a collaboration between Theia Scientific, LLC and Volkov Labs, utilizes AI and machine learning to automate image analysis and data quantitation, providing real-time observability of experiments. It integrates multiple data sources and plugins, offering customizable visualizations and interactive panels, such as an Acquisition panel for controlling image sources and a Vision panel for viewing AI analysis results. This setup allows scientists and engineers to optimize their experimental parameters and immediately see the effects, enhancing their ability to conduct experiments and analyze data efficiently. The dashboard's success is attributed to its adaptability and the use of Grafana's ecosystem, and there are plans to further develop its capabilities by incorporating dynamic dashboards and additional observability features to support the evolving needs of scientific research.

Grafana 10 introduces Grafana Scenes, a new frontend library aimed at enhancing the dynamic dashboarding capabilities of the platform by enabling developers to create more interactive and customizable experiences within their Grafana application plugins. This TypeScript library facilitates the embedding of comprehensive dashboards with template variables, time ranges, and visualizations, allowing developers to incorporate dynamic features like conditional panels and user-driven interactions. While it is currently tailored for plugin developers familiar with TypeScript, there are plans to extend its accessibility to non-plugin users through more declarative models. Grafana Scenes is in public preview across various Grafana offerings, and its development roadmap includes achieving feature parity with existing dashboards and exploring new functionalities.

Grafana Tempo 2.2 introduces significant enhancements to its TraceQL query language, notably the addition of structural operators which allow users to efficiently search for complex conditions in distributed traces, making debugging and trace analysis more effective. This release also introduces new trace-level intrinsics for improved query efficiency, a select() operator for low-overhead field surfacing, and a by() operator for splitting spansets by specific fields, enhancing the flexibility and performance of trace queries. Additional updates include support for ipv6, a new API for configuring limits, and span filtering in metrics generators, alongside experimental features like a dynamic metrics API from tracing data and results streaming for TraceQL. These updates reflect Grafana's ongoing commitment to enhancing Tempo's performance and functionality, with future plans to expand TraceQL features and improve the handling of dynamic metrics. The release underscores Grafana's focus on building a powerful and efficient distributed tracing tool, encouraging community engagement and feedback to shape future developments.

Grafana Cloud offers a fully managed observability platform that simplifies the setup and configuration of its open-source software, providing users with a comprehensive stack for logs, visualization, traces, and metrics through its free and paid tiers. During GrafanaCON 2023, the company announced an expansion of its Cloud Free tier to include access to all Enterprise plugins, which were previously exclusive to paid plans, alongside features like load testing, incident management, and continuous profiling. Kyle Shelton, a Senior DevOps Engineer, shared his personal experience of using Grafana Cloud Free to track his triathlon training, highlighting its ease of use and the ability to visualize diverse data. The platform's integrations and prebuilt solutions facilitate quick deployment and analytics, which Shelton leveraged to manage his data effectively despite time constraints as a new parent. Grafana Cloud is accessible on major cloud platforms such as AWS, Microsoft Azure, and Google Cloud, with additional partnerships in the Asia-Pacific region. Users are encouraged to share their unique experiences with Grafana Cloud Free, showcasing its practical benefits in various settings.