June 2023 Summaries

Ultimate, a company specializing in customer support automation, significantly enhanced its incident response management (IRM) by transitioning to Grafana Cloud, which streamlined workflows, improved adoption, and facilitated better communication among its 20-person engineering team. Previously, their IRM processes involved juggling multiple tools, leading to inefficiencies and a high cognitive load for developers. By adopting Grafana OnCall, Ultimate centralized its incident alerts, allowing seamless integration with dashboards and communication platforms like Slack, thus ensuring consistent incident management and timely updates. The migration also encouraged broader company adoption, including non-technical teams like Customer Success, who began independently creating alerts and contributing to system monitoring. The shift to Grafana Cloud not only reduced tool overhead and costs but also fostered a culture where teams were empowered to build and manage their own monitoring solutions, thereby enhancing overall system observability and efficiency. Looking ahead, Ultimate plans to further expand its use of Grafana’s observability tools, aiming to improve capacity planning, performance analysis, and reduce mean time to recovery (MTTR).

Grafana 10, showcased at GrafanaCON 2023, marks a significant upgrade in the observability tool's capabilities, celebrating a decade since the launch of Grafana OSS. This latest release introduces a user-friendly experience aimed at both new and experienced users, with enhancements in navigation, plugin development tools, and data visualization options, such as the Canvas panel for custom data representation. Public dashboards have been updated with added security features, while the new Scenes framework provides a versatile JavaScript-based approach for creating dashboard-like experiences. The release also focuses on simplifying the process for developers creating plugins, including a new toolkit and automated compatibility checks to streamline plugin development and publishing. As Grafana expands its user base beyond developers to include business and financial analysts, it continues to refine its platform to accommodate diverse data visualization needs, making it more accessible and powerful for a broader audience.

Grafana Labs has introduced a new feature called the metrics explorer to enhance the Prometheus query builder experience for users of varying expertise levels. This feature, available in Grafana 10.1, aims to simplify the process of building PromQL queries by offering a paginated and searchable interface for browsing metrics, allowing both novices and advanced users to construct more effective queries. The metrics explorer enables fuzzy searches by name or description, filters metrics by Prometheus types, and displays comprehensive metric information, including name, type, and description, in a paginated list. It also includes an expert feature for regex searches, offering flexibility and depth in query building. This initiative is driven by the need to make Prometheus, a standard for monitoring applications, more accessible and user-friendly, especially for those unfamiliar with its extensive metric options. As Grafana Labs continues to refine this tool, they are seeking feedback from the community to enhance its functionality further.

Grafana, a popular open-source platform for data visualization, transitioned from Angular to React in 2018 to leverage React's extensive ecosystem, performance optimizations, and widespread use, which have facilitated faster feature development and community contributions. However, the team faced challenges with React, particularly in state management, requiring a mix of tools like Redux, RxJS, props, and context, and the complexity introduced by React's hooks paradigm. To address these issues, Grafana developed a new dashboard runtime library called Scenes, which streamlines the development process by minimizing reliance on complex hooks and enhancing code clarity. Despite the challenges, React remains a valuable tool for Grafana, and the ongoing exploration of new methodologies and tools like Scenes aims to further improve the development experience.

Grafana k6 v0.45.0 introduces several new features and improvements, including an experimental module for gRPC streaming support and an updated browser recorder extension for Firefox and Chrome. Users can now upload test scripts to Grafana Cloud without executing them, which is particularly useful for CI processes. Additionally, the release includes updates to the k6 browser, such as replacing existing metrics with Google Web Vital metrics and fixing bugs related to timeout errors and remote object parsing. Grafana Cloud k6 now features a browser recorder for generating scripts from sessions and a project activity view, along with enhanced notification options and security monitoring capabilities. The k6 community continues to expand with new contributors and champions, who actively participate in conferences and write articles on load testing strategies. The release cycle also saw updates to various extensions, including support for the gRPC protocol and improvements to the xk6 output and security features. Users are encouraged to engage with the k6 community via multiple platforms and to take advantage of the free-forever tier offered by Grafana Cloud for performance testing.

Grafana Loki is a crucial component of the Grafana Labs LGTM stack, designed for efficient log aggregation as part of a comprehensive observability solution alongside Grafana, Tempo, and Mimir. With its ability to scale horizontally, support multi-tenancy, and integrate seamlessly with popular storage backends, Loki is tailored for developers and operators to enhance their logging capabilities. Grafana Labs provides a range of resources, including webinars and GrafanaCON on-demand sessions, to help users at various expertise levels explore Loki’s features—from getting started and configuring for high availability to optimizing query performance with the new TSDB index. Advanced use cases, like NVIDIA's Threat Detection System, highlight Loki's effectiveness in large-scale security log analysis, underscoring its open-source nature, cost-effectiveness, and flexibility.

Kubernetes cost monitoring in Grafana Cloud is a newly introduced feature designed to help organizations manage and optimize the costs associated with their Kubernetes infrastructure by providing real-time insights and detailed visualizations. Built on the open-source tool OpenCost, it allows users to track current and historical spending, identify resource-intensive components, detect anomalies, and make informed decisions about resource allocation and scaling. This feature benefits site reliability engineers, administrators, and engineering leaders by enabling granular cost control, improving resource efficiency, and facilitating cost governance practices. By integrating cost monitoring with existing dashboards in Grafana Cloud, organizations can personalize their Kubernetes ecosystem while enhancing cost efficiency and return on investment.

Grafana Cloud has introduced a new integration specifically for monitoring Apache CouchDB, a NoSQL document database known for its scalability and easy data replication. This integration leverages CouchDB's Prometheus metrics endpoint and the Grafana Agent to simplify monitoring, allowing users to track various metrics such as database reads/writes, request latencies, and replication states. By setting up a Grafana Cloud account, users can access pre-built dashboards and alerts that provide insights into the health and performance of their CouchDB clusters and nodes. The integration includes two main dashboards: the Apache CouchDB Overview, which offers a high-level view of clusters, and the Apache CouchDB Nodes, which provides detailed node-specific metrics. Additionally, it features 10 alerting rules to monitor cluster health, request status codes, and replication status. This integration aims to make CouchDB monitoring accessible and efficient, aligning with Grafana Labs' philosophy of offering a consistent user experience across diverse data sources and environments.

Grafana 10 introduces Private Data Source Connect (PDC), a feature that allows Grafana Cloud users to securely monitor and query data sources within private networks, addressing the challenge of network security that arises when connecting to databases like MySQL or Elasticsearch within on-prem infrastructures or virtual private clouds. PDC, now generally available to all Grafana Cloud users, including those on the free tier, provides a secure connection via SOCKS over SSH, enabling users to deploy a lightweight agent on their network to connect with Grafana Cloud while maintaining control over the network connection. This feature, announced at GrafanaCON 2023, is easy to set up and manage, supporting industry-standard security protocols and working across various public cloud vendors. Users can connect their Grafana Cloud stack to one private network with multiple data sources, and while currently limited to a subset of data sources, future developments will expand its capabilities.

Grafana has released several new versions, including 10.0.1, 9.5.5, 9.4.13, 9.3.16, 9.2.20, and 8.5.27, to address a critical security vulnerability identified as CVE-2023-3128, which affects Azure Active Directory (AD) account validation. This flaw, with a CVSS score of 9.4, can lead to account takeover and unauthorized access to sensitive data when Azure AD OAuth is configured with a multi-tenant application. Users are advised to upgrade their instances or apply mitigations like configuring allowed_groups or registering a single tenant application in Azure AD. The patch removes unsafe email lookups and relies on unique user IDs, which may disrupt authentication workflows in certain scenarios. Grafana Labs emphasizes not disclosing vulnerabilities until they are officially announced and provides channels for reporting security issues.

Rivanor Soares, Manager of Platform Observability at Just Eat Takeaway, emphasizes the importance of collaboration and inclusivity in his team of engineers, who manage observability tooling across diverse environments. He describes himself as a people-first manager who ensures every team member feels heard and included, fostering a supportive environment where team members readily help each other. Soares seeks candidates with a deep understanding of cloud services and the ability to summarize technical experiences from a practical perspective during interviews. He values transparency and honesty, preferring to consult his team for technical expertise rather than providing incomplete answers. Soares also prioritizes candidates who demonstrate confidence, creativity, and the humility to admit when they do not know something, while appreciating those with experience in development languages relevant to the company's needs. The success of team members is measured by their ability to complete projects efficiently and contribute new ideas, with different expectations set for senior and intermediate members. Just Eat Takeaway is a global online food delivery marketplace headquartered in Amsterdam, connecting consumers with a wide variety of restaurant partners.

Grafana Labs has developed an integration to monitor TensorFlow Serving, a widely-used machine learning model server, through Grafana Cloud. TensorFlow Serving simplifies the deployment of AI models by offering features like model versioning and support for canary deployments. The integration leverages TensorFlow Serving's built-in Prometheus metrics to generate telemetry data, allowing users to monitor their AI environments with ease. By installing this integration, users gain access to pre-built dashboards and alerts that track key performance metrics such as model request rates, latency, and batch queue throughput. Alerts are triggered for high error rates and queuing latency, with customizable thresholds to suit different environments. Grafana Cloud offers a free tier and provides a consistent experience across various data sources, supporting a wide range of monitoring needs.

Grafana Cloud, a composable observability platform, offers a free tier that includes up to 10,000 active series and is billed based on usage, prompting questions about costs associated with monitoring different numbers of servers. The platform supports multiple data sources, focusing on Graphite and Prometheus for estimating time series requirements. With Hosted-Graphite, users can monitor approximately 33 servers under the free tier, assuming 300 series per server, whereas Prometheus, which uses more detailed metrics, supports around 13 servers, with node_exporter exporting about 700 active series per server. Users needing more capacity can upgrade to Cloud Pro or Cloud Advanced plans. Grafana Cloud's fully managed LGTM Stack, which includes Grafana Loki, Grafana Tempo, and Grafana Mimir, provides users with comprehensive observability without the hassle of infrastructure management, and allows integration with other tools for metrics, logs, and traces. Additional capabilities include performance testing, incident management, and infrastructure monitoring, with webinars available for further learning about the platform's features.

Grafana, an open-source data visualization tool created by Torkel Ödegaard, has reached significant milestones since its inception in 2013, evolving from 12,000 instances at its first GrafanaCON in 2015 to over 1.2 million active instances and an estimated 20 million users by 2023. Ödegaard reflects on key moments throughout the project's journey, such as the enthusiastic early adoption by the monitoring community, the decision to work on Grafana full-time following positive reception at Monitorama 2014, and the excitement of seeing Grafana used in high-profile settings like SpaceX's Falcon 9 control center. The platform has introduced numerous innovative features over the years, including gradient color scales, state timeline panels, and query builders, which have set it apart in the data visualization space. Grafana's design philosophy emphasizes simplicity and a clean aesthetic to ensure data is prioritized, a vision that has been embraced by a supportive open-source community. As Grafana celebrates its tenth anniversary with the release of Grafana 10, a documentary chronicling its journey is set to be released, highlighting its evolution and impact on data visualization.

Grafana Labs recently celebrated its 10th anniversary with GrafanaCON 2023, launching Grafana 10 and introducing the Golden Grot Awards to honor exceptional Grafana dashboards developed by its community over the past decade. The grand prize winners in the personal and professional dashboard categories were Nicky Sonnemans from the Netherlands and Raymond Sowden from South Africa, respectively. Nicky's dashboard focuses on the European energy infrastructure, offering detailed insights into gas storage and flow, electricity generation, and energy prices using a tech stack that includes Airflow and Grafana. Raymond's dashboard provides a comprehensive overview of system health for JD Edwards EnterpriseOne ERP, allowing users to drill down into detailed metrics and reducing the need for manual system checks. Both dashboards were praised for their ability to transform complex data into accessible, visually appealing formats that provide valuable real-time insights.

Day two of GrafanaCON 2023 was filled with exciting updates and demonstrations, focusing on the newly released Grafana 10 and its accompanying features, such as Grafana Scenes and enhancements in Grafana Tempo and Mimir. Attendees delved into various applications of Grafana, from optimizing home automation systems using Raspberry Pi and Grafana Cloud to managing real-time telemetry data at Daimler Truck with the Grafana Stack. The event also showcased the use of Grafana in performance testing with k6 and its role in evolving API strength. Sessions highlighted the improvements in Grafana's documentation, navigation, and user experience, underscoring the continuous efforts to align Grafana with the open-source observability ecosystem. Additionally, participants explored how to leverage Grafana for data-driven pursuits like triathlon training, emphasizing its versatility for both personal and industrial applications.

GrafanaCON 2023's first day showcased a range of advancements and insights in the realm of open-source analytics and monitoring tools, with the spotlight on the release of Grafana 10, updates to Grafana Loki, and various community-driven innovations. The keynote address by Torkel Ödegaard and Mihaela Maior highlighted the new features of Grafana 10, while other sessions explored its application in diverse fields such as IoT monitoring, green IT initiatives, and satellite tracking with the Libre Space Foundation. Notably, Grafana dashboards were demonstrated as vital tools for real-time data monitoring and decision-making, evidenced by DHL Express Switzerland's improved IT workflows and Sentry Software's energy-saving measures. The introduction of Grafana Pyroscope for continuous profiling and Grafana Agent Flow for telemetry collection underscored Grafana's commitment to enhancing user experience and operational efficiency. The event emphasized the transformative potential of Grafana in fostering a high-performance data culture and its adaptability across various use cases, including NVIDIA's security log analysis and ESnet's network data visualization.

GrafanaCON 2023 served as a milestone event for Grafana Labs, marking the official release of Grafana 10 and celebrating a decade of the platform's growth, characterized by its widespread adoption and innovative visualizations. Co-founders Raj Dutt, Torkel Ödegaard, and Anthony Wood reflected on Grafana's journey from its inception in 2013, highlighting advancements in the Grafana ecosystem, which now boasts 20 million users and 150 data sources. Grafana 10 introduces enhanced visualizations, improved UI for easier dashboarding, new security features, and a preview of Grafana Scenes for advanced dashboard customization. The event also announced expansions to Grafana Cloud's free tier, offering more enterprise plugins and tools for observability and performance testing. Additionally, the Golden Grot Awards recognized outstanding dashboards from the community, celebrating the platform's impact and creativity. The event included 30+ sessions and local meetups worldwide, providing opportunities for community engagement and learning.

Grafana 10 introduces significant updates to enhance user experience, data visualization, and security for its global user base exceeding 20 million. The release features new visualization panels, such as the trend and datagrid panels, which enable dynamic and customizable dashboards to facilitate real-time data insights. Grafana 10 also introduces the Correlations feature to unify data from multiple sources, enhancing data analysis capabilities. The platform now supports improved public dashboard sharing and introduces Grafana as code, optimizing development with enhanced plugins and development tools. Additionally, the release includes updates for better navigation and onboarding, such as a new onboarding experience and a reorganized menu to streamline data retrieval and dashboard management. Security enhancements include the Private Data Source Connect for querying private data securely and a simplified SAML authentication setup to ensure fine-grained access control. These advancements aim to empower users to derive more insights, foster collaboration, and make confident data-driven decisions.

The Grafana OnCall mobile app, available on Google Play and the Apple App Store, enhances the experience of on-call management for engineers by providing a personalized and integrated mobile solution. Built on the Flutter framework, it syncs with both the open-source and cloud-hosted versions of Grafana OnCall, allowing users to manage alerts, configure notification policies, resolve alert groups, and check schedules directly from their devices. The app addresses common issues faced by on-call engineers, such as disturbances during critical alerts in Do Not Disturb mode, by allowing alerts to break through these settings without affecting other notifications. Users can customize notification preferences and receive timely reminders before shifts, with the app further aiming to reduce stress through carefully designed alert sounds. Future updates will include additional features like on-call reporting and deeper integrations with Grafana's incident response management tools, with users encouraged to participate in the open beta for early access and feedback.

GrafanaCON 2023 is set to take place virtually from June 12 to June 14, featuring over 30 live sessions on topics including homelab projects, IoT monitoring, and the latest features in Grafana 10. The event will also highlight updates to the Grafana LGTM Stack, which includes Loki, Grafana, Tempo, and Mimir, and discuss best practices for other open-source projects like Grafana k6 and Pyroscope. As Grafana Labs celebrates 10 years of its open-source project, attendees can participate in free workshops to improve their dashboard skills, engage in live Q&A sessions, and network with the community via the Grafana Labs Community Slack workspace. The event will begin with an opening keynote and the presentation of the inaugural Golden Grot Awards for outstanding Grafana dashboards. Sessions will be available on-demand after each day, and those interested can also attend local meetups in over 20 cities worldwide.

Grafana has announced the public preview of its Cloud Logs Export feature, which allows Grafana Cloud users to export logs to their own cloud-based object storage solutions, such as Amazon S3, Google Cloud Storage, and Microsoft Azure Blob, for long-term archival at no additional cost. This feature is designed to address customer needs for extended log retention due to compliance requirements and the desire for cost-effective data control. Users can configure the export functionality through the Grafana UI, selecting their preferred cloud provider for storage, and can query the exported data using LogCLI or a self-hosted version of Grafana Loki. The new feature aims to provide an affordable solution for those needing to retain logs longer than the default 30-day period in Grafana Cloud, while offering more autonomy over their data management and storage. Users across all Grafana Cloud tiers, including the free tier, can access and test this feature, which is particularly useful for those with compliance and audit requirements, as well as those wishing to conduct downstream analytics on archived logs.

Grafana Agent v0.34 introduces several enhancements, including improved Kubernetes monitoring and support for HashiCorp Vault, with a focus on increased community involvement. The release advances feature parity between Grafana Agent Flow and its static mode through additional exporters, such as those for Snowflake and Microsoft SQL Server, and introduces the remote.vault component to handle sensitive information securely. It further enhances Kubernetes integration with the prometheus.operator.servicemonitors component, allowing for the discovery and scraping of ServiceMonitors, and incorporates community contributions such as new exporters and updates to the River standard library. The release underscores Grafana’s commitment to a vendor-neutral approach, enabling users to work with familiar protocols and formats without requiring application changes.

Grafana Cloud has introduced a new monitoring solution specifically for Docker Desktop, which is a popular tool among developers for containerization on local machines. This solution allows users to monitor metrics and logs for containers running within Docker Desktop by using an embedded Grafana Agent. The integration uses metrics generated by the open-source cadvisor exporter for the Docker engine, simplifying the process of collecting data with a single agent and configuration. To get started, users need Docker Desktop version 4.8 or higher and can install the Grafana Cloud extension from the Docker Desktop extensions marketplace. Once installed, the integration provides three prebuilt dashboards in Grafana Cloud, offering an overview of Docker Desktop metrics, logs, and node-specific data. Additionally, the embedded Grafana Agent can monitor other systems like MySQL within Docker Desktop containers. This integration is designed to simplify the setup of monitoring solutions and enhance data visibility, with a free tier available for new users.

Grafana Cloud has announced the public preview of Grafana Cloud Profiles, a significant development that integrates the Pyroscope and Phlare open-source projects, enhancing continuous profiling capabilities. This launch, available to both free and paid users, offers 50GB of profiles with a 14-day retention period. The integration makes Pyroscope horizontally scalable, accommodating a variety of workloads, and standardizes its SDKs to support major programming languages, including Go, Python, and Java, among others. The platform incorporates insights from community feedback to provide versatile visualization options such as an app plugin, explore view, and dashboard view, each designed to improve the analysis and visualization of profiling data. Grafana Labs is committed to enhancing the profiling experience, emphasizing its potential to boost organizational efficiency and competitive advantage. Users are encouraged to provide feedback, which will influence the development of Grafana Cloud Profiles and the Pyroscope project, with an OSS Pyroscope 1.0 release anticipated soon.

Grafana Labs has released new versions 9.5.3, 9.4.12, 9.3.15, 9.2.19, and 8.5.26 to address security vulnerabilities CVE-2023-2183, which involves broken access control allowing a Viewer role to send test alerts, and CVE-2023-2801, a data source query vulnerability that can crash a Grafana instance. The patches have been applied to Grafana Cloud, and major cloud providers like Amazon and Azure have ensured their versions are secure. The CVE-2023-2183 vulnerability has a medium severity score of 4.1, while CVE-2023-2801 is considered high with a score of 7.5 when public dashboards are enabled. To mitigate these vulnerabilities, users are advised to upgrade their Grafana instances and apply specific patches to prevent mixed query concurrent calls. Grafana encourages reporting security issues via their dedicated email, where security vulnerabilities are to be encrypted with their PGP key for privacy.

Grafana Incident, an incident response tool from Grafana, has expanded its capabilities with new APIs and webhooks that facilitate seamless integration into diverse workflows. These enhancements enable teams to customize their incident response management processes by connecting Grafana Incident with a wide range of external systems, including Zoom, GitHub, and ServiceNow, among others. The JSON/HTTP RPC API allows detailed programmatic control over incident activities, timelines, and tasks, while outgoing and incoming webhooks automate workflow triggers based on specific events. Several use cases illustrate these capabilities, such as posting updates to incidents, triggering workflows in third-party systems like ServiceNow, and automating escalation processes based on incident labels. These features aim to improve efficiency, collaboration, and communication within engineering teams by automating task management, notifying relevant parties promptly, and enhancing incident detection and response times. The integration options provided by Grafana Incident's APIs and webhooks support the creation of tailored workflows that align with the unique needs of different teams, ultimately streamlining incident management processes.

GrafanaCON 2023 is hosting a series of local meetups in over 20 cities globally throughout June, coinciding with its virtual event from June 12 to 14, to foster community connections and provide in-depth discussions on the latest release, Grafana 10. These two-hour in-person events, mostly hosted at AWS offices due to a partnership with Amazon, will feature presentations, Q&A sessions with Grafana experts, and networking opportunities for attendees. Registration is open for many locations, and detailed agendas are available. For those unable to attend in person, virtual sessions of GrafanaCON 2023 include talks on various topics such as homelab projects and IoT monitoring. Grafana Labs offers additional opportunities for connection through upcoming webinars and conferences.

Grafana k6 browser and Google Lighthouse are tools used to assess web performance, each with distinct functionalities and methods of testing. The k6 browser module, part of Grafana k6 OSS, simulates user interactions to gather frontend performance metrics, allowing for a hybrid testing approach that combines browser and protocol-level scripts. It provides insights into user experience by measuring metrics such as First Input Delay (FID), which requires real user interaction. Conversely, Google Lighthouse is an open-source tool that evaluates overall website quality, including performance, accessibility, and SEO, generating audits and suggestions for improvements. Lighthouse primarily uses lab data, making it ideal for early diagnostics, but it doesn't measure FID due to its focus on initial page load analysis. Both tools report on Google Web Vitals, though Lighthouse additionally tracks metrics like Speed Index. Despite their differences, combining the use of both tools with additional field data sources, such as PageSpeed Insights, enhances the accuracy of web performance evaluations by capturing real user experiences.