September 2023 Summaries

Microsoft Azure is retiring Azure Active Directory (AAD) Pod Identity in favor of Azure AD Workload Identity (WI) authentication as of October 1, 2023, prompting Grafana Labs to collaborate with Microsoft to update Grafana for compatibility with WI authentication. Users are encouraged to upgrade to the latest versions of Grafana, specifically 10.1, 10.0, or 9.5, to ensure seamless integration with Azure resources via WI authentication and avoid issues with data sources like Azure Monitor and Prometheus that were previously configured with AAD authentication. Detailed instructions for configuring WI authentication are available in the Grafana documentation, and users can seek further assistance through the Grafana Labs Community Slack or their standard support channels.

Grafana and Grafana Enterprise have temporarily halted updates for ARMv6 and ARMv7 due to a bug in the Go compiler that specifically affects 32-bit ARM versions. This issue, which emerged earlier this year, has been challenging to resolve due to its sensitivity to binary size and code layout, affecting the continuous integration process for building Grafana. The company has escalated the problem to the Go team and is exploring alternative solutions while ensuring that existing ARMv6 and ARMv7 builds remain stable and accessible. Users can still access the latest stable versions from the Download Grafana page, and ongoing updates about the situation are being provided through a dedicated Slack channel for community support.

Grafana has introduced a new feature called "shift swaps" in Grafana OnCall, designed to streamline the process of exchanging on-call shifts among software engineers, thus accommodating personal events and unexpected plans without the chaos of manual coordination. Available in both the open-source and hosted versions of Grafana OnCall, as well as its mobile app, this feature allows users to initiate and accept swap requests through a user-friendly interface, with notifications facilitated via Slack or the Web UI. If no one accepts a swap request, the original schedule remains unchanged, but reminders are sent to ensure visibility of pending requests. This feature, distinct from the existing "override calendars" which lack automatic notifications, is part of a series of updates aimed at enhancing the Grafana OnCall experience, alongside new scheduling and email support features. Grafana OnCall is integrated within the Grafana IRM suite, available through Grafana Cloud, which offers a free tier for users to explore its functionalities.

The Prometheus Java client 1.0.0 release marks a significant update to the metrics library, enabling Java applications to be instrumented with Prometheus metrics, and introducing new features such as native histogram support and OpenTelemetry integration. This version represents a complete rewrite of the data model and API, enhancing performance, interoperability, and configurability. The update aligns with emerging open standards, allowing seamless integration with OpenTelemetry and improving runtime configuration through various options. Extensive documentation and end-to-end examples are now available, addressing previous shortcomings in user guidance. The update aims to support high-performance applications by optimizing both metric types and operations for concurrent use, with future plans to expand module support and enhance integration with OpenTelemetry's semantic conventions. Additionally, Grafana Labs continues to support Prometheus with a range of dashboards, enhancing the monitoring setup with compliance to semantic conventions.

In OpenTelemetry metrics, the choice between Delta and Cumulative temporality involves significant trade-offs related to data reporting and backend compatibility. Delta temporality records changes between reporting intervals, while Cumulative temporality aggregates data from the start of the measurement. The decision on which temporality to use often hinges on the capabilities of the backend systems, with some, like Prometheus, favoring Cumulative due to its resilience against data loss from dropped samples. The OpenTelemetry Collector can convert between temporalities, but scaling such conversions poses challenges, particularly when addressing issues like out-of-order writes and data aggregation. Cumulative temporality typically requires more memory for Synchronous Instruments due to storing every metric-label combination, whereas Delta temporality can minimize memory usage and reduce the risk of memory leaks. The low-memory mode of OpenTelemetry SDKs balances this by using Delta for Synchronous and Cumulative for Asynchronous Instruments. Ultimately, the best practice is to align temporality with backend support, with Cumulative often recommended for its robustness against data gaps and simplified pipeline management.

Grafana Labs celebrates a decade of innovation and community engagement, highlighting the top 10 most notable Grafana dashboards created by its users worldwide. The open-source visualization tool, which now boasts over 20 million users, has been utilized in diverse and groundbreaking ways, from monitoring SpaceX rockets and Microsoft's underwater data centers to enhancing virtual experiences in Call of Duty and real-world events like the Tour de France and The Ocean Race. Co-founders Raj Dutt and Anthony Wood express gratitude for the community's role in Grafana's success, noting its widespread adoption in everything from office spaces and video games to major scientific and sporting events. As Grafana marks its 10-year anniversary, the founders reflect on the tool's impact and anticipate future advancements, inviting the community to share their stories and join upcoming events like ObservabilityCON 2023.

Daimler Truck utilizes an advanced observability stack to manage its tb.lx service, which processes high-volume telemetry data from connected vehicles in real time. This stack is built on Grafana, Grafana Loki, Prometheus, and Pyrra, and is designed to maintain high availability and low latency. Principal Engineer Adrien Bestel highlighted their approach in a GrafanaCON 2023 talk, explaining the four-step process his team used to define, implement, and monitor service level objectives (SLOs). These steps involve building a vendor-neutral observability stack with open-source tools, defining SLOs and error budgets for key performance indicators like availability and latency, implementing these SLOs using Pyrra and Kubernetes, and safeguarding them through continuous monitoring and testing using Grafana dashboards. This approach helps the team maintain clear baselines for performance improvement, ensuring that SLOs are met and error budgets are managed effectively, which ultimately contributes to customer success.

Grzegorz Piechnik, a performance engineer and k6 Champion, discusses the challenges and solutions in automating anomaly detection in system observability and performance testing. Throughout his career, he encountered barriers to automation, particularly in test result analysis, which led him to develop the xk6-anomaly extension. This tool enhances the Grafana k6 platform by enabling the automated detection of anomalies, such as outliers and event shifts, in performance test data. By leveraging statistical techniques and machine learning algorithms, xk6-anomaly identifies deviations from expected patterns, thus improving efficiency and allowing engineers to focus on other critical tasks. The extension works by analyzing metrics extracted from k6, integrating seamlessly into test scenarios, and offering flexibility in detecting relevant anomalies across various environments. This innovation has significantly streamlined Piechnik's workflow, transforming the manual and monotonous process of anomaly detection into a more satisfying and productive task.

The concept of "learning in public" is explored as a method for accelerating learning and supporting open-source software (OSS) communities by sharing knowledge and experiences online. This practice involves openly discussing one's learning journey, including mistakes and uncertainties, to foster community engagement and trust. It leverages the collective expertise of the community to enhance individual learning and contributes to the broader knowledge base, as exemplified by technical writer Julia Evans. Despite its simplicity, learning in public requires courage, as it can trigger imposter syndrome and fear of being wrong. However, embracing this approach can transform learning experiences by encouraging open dialogue, correcting errors, and accessing communal knowledge. The text also contrasts this method with being a silent student or projecting an expert facade, both of which can limit learning opportunities. The author invites discussion and feedback, aligning with Grafana Labs' values of collaboration, transparency, and supporting OSS.

Universities are utilizing Grafana dashboards to enhance the monitoring and preservation of digital assets through a distributed digital preservation network, SAFE PLN, which involves seven partner universities across Europe and Canada. This network is designed to safeguard scientific publications, theses, and research data by using LOCKSS technology, developed at Stanford University, to replicate and verify data integrity across multiple nodes. The adoption of Grafana, alongside Grafana Loki and Prometheus, has streamlined the monitoring process by providing real-time visibility and alert systems, replacing a previously cumbersome custom application. This setup allows for immediate detection and response to network issues, thereby improving security and efficiency significantly. The initiative has proven its resilience and adaptability, prompting adoption by other LOCKSS networks, including the Alabama Digital Preservation Network and the MetaArchive Cooperative, indicating a growing trend in utilizing Grafana for digital preservation efforts.

Grafana Labs has introduced an agentless integration for Prometheus metrics into Grafana Cloud, called the Metrics Endpoint solution, which allows users to forward metrics from any internet-accessible endpoint without the need for additional infrastructure. This solution aims to reduce the operational overhead associated with deploying agents like Grafana Agent, making it particularly appealing for SaaS users who may not have dedicated infrastructure. Unlike traditional agent-based integrations, this agentless approach is designed for metrics accessible over the internet, offering a seamless experience similar to existing agentless integrations such as AWS CloudWatch and Confluent Cloud. Developed from a hackathon initiative, the Metrics Endpoint solution enables users to create scrape jobs with configuration details to access various data sources, exemplified by its application with Supabase, where users can easily set up monitoring by using basic authentication credentials. This integration aligns with Grafana's commitment to providing a consistent and expansive observability experience across diverse environments, and it is now part of Grafana Cloud's offerings, including its free-tier access with additional Enterprise plugins for three users.

ObservabilityCON 2023 is set to take place in London from November 14-15, focusing on the latest trends in open source observability and updates to the Grafana LGTM Stack, which includes Loki for logs, Grafana for visualization, Tempo for traces, and Mimir for metrics. Attendees can participate in technical workshops, hear keynotes from Grafana Labs executives, and engage in sessions covering topics such as cost management with Grafana Cloud, best practices for correlating application and infrastructure observability, and the role of AI in observability. The event will also feature success stories from the Grafana community and provide networking opportunities, including an "Ask the Experts" booth. Registration is open, with early bird pricing available until October 20, and a live stream of the keynote will be available for those unable to attend in person. Additionally, Grafana Labs is conducting an observability survey to gather insights from the community.

Distributed Tracing in Grafana Cloud k6, now generally available, bridges the visibility gap between performance testing and system tracing, allowing engineering teams to troubleshoot failed performance tests more efficiently. By integrating Grafana Cloud Traces with Grafana Cloud k6, users can correlate performance test results with server-side tracing data, reducing the mean time to recovery (MTTR) and improving application reliability. This integration enables automatic injection of tracing metadata into backend service requests, supporting major propagators like W3C (OpenTelemetry) and Jaeger. The collected data generates real-time metrics and provides a summary view of all spans generated during testing. This allows users to identify root causes of issues, such as in a hypothetical scenario where a locking issue in MySQL caused significant latency. Users can explore specific traces to diagnose problems, making Distributed Tracing in Grafana Cloud k6 a valuable tool for proactively addressing potential customer and revenue impacts. The integration is available for all Grafana Cloud users, including those on the free tier, and setup involves sending service tracing data to Grafana Cloud Traces and enabling the tracing feature in Grafana Cloud k6 tests.

Grafana OnCall has been enhanced to better support enterprise-scale incident response management by introducing features like role-based access control (RBAC), cross-team linking, a PagerDuty migration tool, and advanced webhook functionality. These improvements aim to address the unique challenges faced by large organizations, such as managing permissions, linking team schedules, and integrating with existing workflows. Grafana OnCall now allows for seamless linking of different team objects to automate cross-team escalations, and the PagerDuty migrator simplifies the transition process for users switching platforms. Additionally, the new Grafana OnCall Insights Logs and Metrics offer predefined parameters for tracking telemetry data and understanding on-call dynamics without setup for Grafana Cloud users. Enhanced webhook capabilities facilitate deeper integration with tools like ServiceNow, allowing automatic incident management. These upgrades are part of Grafana's broader incident response management offering, which includes Grafana Incident, and are available through Grafana Cloud's expanded free tier.

Prezi, a video and visual communications company, undertook a strategic shift to Grafana's Incident Response & Management (IRM) suite, replacing their long-used PagerDuty system to enhance efficiency and reduce costs. The transition, driven by the integration of Grafana Cloud for log management, led to the adoption of Grafana Alerting, Grafana OnCall, and Grafana Incident, offering a seamless workflow and cost-effectiveness. The Prezi Site Reliability Engineering (SRE) team conducted a detailed comparison between the two platforms, focusing on essential features such as alert collection and escalation, and found that while both systems offered similar functionalities, Grafana IRM presented significant annual savings. The migration process involved a lift-and-shift approach, with existing configurations duplicated for IRM, enabling a swift transition without disrupting ongoing operations. Despite encountering challenges such as unused configurations and limitations with the Terraform provider, the Prezi team successfully integrated IRM into their infrastructure, marking the switch as a successful cost-saving alternative to PagerDuty.

Grafana Labs has introduced Sift, an automated diagnostic feature in Grafana Cloud's Incident & Response Management suite, designed to enhance incident response times by performing system checks and identifying potential issues within Kubernetes environments. Sift leverages Grafana Machine Learning and integrates with the Grafana LGTM Stack to utilize metrics, logs, and traces for automating routine incident investigations, such as analyzing error patterns, Kubernetes crashes, resource contention, and slow requests. By providing insights into problems like overloaded servers or recent service deployments, Sift assists engineers in pinpointing the root cause of incidents more quickly. It can be triggered automatically through Grafana IRM alerts or manually via the Grafana Incident timeline. Currently, Sift is available in public preview for Grafana Cloud users, with plans for expanded capabilities and additional system checks in the future. Users are encouraged to integrate Sift into their incident management processes and provide feedback to aid its development.

ObservabilityCON 2023, hosted by Grafana Labs, is set to take place in London on November 14-15, offering an in-person experience highlighting the latest developments in open-source observability tools, particularly the Grafana LGTM stack. The event will feature keynote announcements, hands-on demos, and deep-dive sessions on industry trends like AI/ML, providing a platform for community members to share insights on solving observability challenges. Interested speakers are encouraged to submit their stories by September 15, with a focus on metrics, logs, traces, SLO management, and more. The event aims to foster a diverse lineup of presenters, including first-time speakers, and promises networking opportunities with Grafana engineers and industry experts from leading companies. Attendees can expect a collaborative atmosphere reminiscent of past events, as highlighted by participants who valued the opportunity for direct interaction with professionals in the field.

Grafana Labs conducted its second annual observability survey to gather insights on the evolving observability landscape, aiming to understand where organizations stand in terms of adoption and market trends. The survey invited participants from various backgrounds, whether they were seasoned with observability concepts or just beginning, to share their experiences in using tools like Grafana, Prometheus, and OpenTelemetry. It sought to capture data on the number and types of tools used, maturity of strategies, and the influence of company size and industry on adoption. Emphasizing the importance of community input, Grafana Labs also offered the opportunity for deeper, anonymized interviews with participants interested in providing more detailed feedback. The survey aimed to take no more than five minutes and ensured anonymity for all respondents, with an incentive of branded merchandise for those participating in interviews. The results, including a contextual analysis, will be published early next year to benefit the wider community.

Grafana Beyla is an open-source eBPF auto-instrumentation tool designed to simplify application observability without requiring code modification or redeployment. Announced at ObservabilityCON 2023, Beyla 1.0 offers auto-instrumentation for Linux HTTP/S and gRPC services, reporting basic transaction span information and RED metrics (Rate-Errors-Duration). It leverages eBPF (Extended Berkeley Packet Filter) to attach programs to the Linux kernel, providing safe, efficient runtime inspections while running in a sandbox environment. Beyla can be set up with Grafana Cloud and integrated with Prometheus for metrics exportation, and it supports various programming languages, including Go, NodeJS, Python, and Java. Although Beyla provides a fast, safe way to gather basic metrics, it does not replace language agents, which offer finer granularity and are recommended for critical operations. Future enhancements include supporting additional protocols and distributed tracing capabilities.

Grafana Scenes, now generally available, is a frontend library designed to enhance the Grafana app development experience by enabling highly interactive and dynamic applications. Initially introduced with Grafana 10 at GrafanaCON 2023 and following a three-month private preview, Grafana Scenes allows developers to build on top of the existing Grafana platform with a declarative API that simplifies creating dashboard features like data querying, time management, and visualization. This library supports familiar Grafana functionalities, such as dynamic pages and drill-down views, while also providing a seamless user experience consistent with the core Grafana product. By drastically reducing the development effort required for common features, Grafana Scenes encourages developers to focus on the core aspects of their applications and introduces new possibilities for creating rich, data-driven solutions. Resources like the Scenes documentation, GrafanaCON 2023 session, and Grafana Office Hours episode are available to help developers get started, with Grafana Cloud offering a free tier for new users.

Grafana 10 introduces a significant upgrade to its log context feature, enhancing log analysis by providing additional context around log entries to better understand incidents or application issues. The update includes a redesigned user interface consistent with the Explore logs visualization, color-coded log lines, integrated timestamps, and a modal view for more focused analysis. Users can now view up to 50 log lines before and after a selected entry and utilize the new Loki log context query editor to refine searches by adjusting label filters or using traceId filters. Future enhancements may include features like remembering selected labels, infinite scroll, and integration with Grafana dashboards for a more comprehensive log view. Grafana encourages feedback on these improvements and offers a free tier on Grafana Cloud for users to start exploring its features.

Grafana Loki, an open-source logging project, reached a milestone of 20,000 GitHub stars as of September 11, 2023, reflecting its significant momentum and community support. Originally introduced by CTO Tom Wilkie at KubeCon Seattle in 2018, Grafana Loki has grown to encompass over 100,000 active clusters and more than a billion Docker Hub image pulls. The project is maintained by a team of 20 and has seen contributions from over 100 developers, resulting in 40 official releases since its first beta in June 2019. The integration of LogQL, a log query language, and features like the TSDB-based index in version 2.7 have enhanced Loki's efficiency, scalability, and usability for log aggregation. RedHat's adoption of Loki alongside Grafana and Prometheus in their OpenShift Kubernetes distribution underscores its utility, while Grafana Cloud employs Loki to manage large clusters for notable clients. As the project evolves, the community is encouraged to participate in workshops and events like ObservabilityCON 2023, where further advancements and announcements are anticipated.

Provisioning notification policies in Grafana Alerting can enhance deployment speed and streamline the notification process by routing alerts using alert labels within a tree structure of nested policies. Users can create or update these policies either through the Grafana UI, configuration files, or an API. While provisioning through configuration files results in non-editable policies in the UI, a workaround exists for API provisioning that keeps policies editable by including an X-Disable-Provenance header. This header allows changes to be made post-provisioning, though it must be included in the initial API request. The process involves retrieving the notification policy tree in JSON format, making necessary modifications, and submitting them via a PUT request. Grafana Alerting supports different platforms, including Grafana OSS, Enterprise, and Cloud, with the latter offering a free tier to get started.

In open source communities, the concept of "good reply game" is crucial for fostering positive interactions and sustaining growth, as it involves responding to others in a manner that supports and expands on their ideas, akin to the "Yes, and..." approach in improv comedy. This practice helps to build shared understanding, friendships, and an inclusive atmosphere, contrasting with "bad reply game," which can lead to nitpicking and hostility, as exemplified by past criticisms of Stack Overflow's community dynamics. By engaging constructively in discussions, participants not only assist the original questioner but also contribute to the broader community, enriching the collective knowledge base and encouraging more dynamic exchanges. The importance of maintaining a good reply game is highlighted as a foundational element of successful OSS communities, where even those who simply ask questions add value, and where respectful and supportive interactions are encouraged, reflecting the core values of organizations like Grafana Labs.

Grafana Loki 2.9 introduces significant improvements including new TSDB volume endpoints for better log volume analysis, enhanced remote rule evaluations for increased reliability, and LogQL optimizations that enhance query performance. The release, which coincides with Grafana Enterprise Logs 1.8, features a restructured documentation system aimed at simplifying navigation and aligning with Grafana's sister teams. Key updates also include support for IBM Cloud Object Storage and multi-store index support, allowing users to specify different storage buckets. These enhancements are designed to improve log exploration, cost management, and overall user experience, and users are encouraged to explore the full list of changes in the Loki 2.9 release notes and changelog.

Grafana Faro Web SDK, when used with Grafana Cloud Frontend Observability, enhances the ability to monitor frontend applications by collecting custom metrics, events, and logs. This integration is particularly beneficial for gaining insights into user interactions on websites and apps, which are often as complex as backend systems. The article explains how to set up and use Grafana Faro to track page speed, errors, and user activities, especially in scenarios where managed services are used, thereby lacking backend access. It highlights the flexibility of creating custom events and metrics for detailed analysis, enabling correlations between frontend data and business metrics, which is crucial for troubleshooting and optimizing performance. The tutorial provided guides users through deploying a basic web application, integrating the Faro Web SDK, and utilizing Grafana Cloud’s tools to visualize and analyze data. This setup allows for proactive monitoring and quick identification of issues, such as detecting a broken "Add to Cart" feature for specific user groups, thereby facilitating timely interventions. The article encourages signing up for Grafana Cloud's free tier to get started with these observability capabilities.

On August 24, 2023, Grafana Labs disclosed a security incident involving the unintentional exposure of their GPG private key and passphrase used for signing packages, which occurred due to a misstep in their CI service's handling of obfuscating secrets. The key exposure was limited to a few internal logs and was quickly addressed by revoking the compromised certificate and issuing a new one. Although the risk of a practical attack was deemed limited, Grafana Labs implemented measures to further secure their CI pipelines by decoupling the signing process from package creation, ensuring the integrity and safety of their distributed packages. The incident was detected internally, and no third-party access or artifact discrepancies were found. A detailed post-incident review and timeline were shared to maintain transparency and emphasize Grafana's commitment to security.

The blog post introduces the Grafana OpenTelemetry Starter, a tool designed to simplify the process of instrumenting Spring Boot applications using Grafana Cloud by integrating recent Micrometer enhancements from Spring Boot 3 with OpenTelemetry. This integration allows developers to consolidate logs, metrics, and traces into a single service, similar to using Prometheus with Spring Boot. The post provides a detailed implementation guide for setting up the Grafana OpenTelemetry Starter in a Spring Boot 3 application, highlighting its role as a Spring starter that facilitates application observability without vendor lock-in. OpenTelemetry, an open-source framework, enables the generation and management of telemetry data, making it easier to switch vendors without significant changes to application code. The guide also includes instructions for configuring the starter, accessing logs and metrics in Grafana Cloud, and utilizing the platform's observability features. Additionally, it offers information on advanced deployment options and community support, encouraging users to explore further resources and sign up for a trial of Grafana Cloud Pro for an enhanced observability experience.

Grafana Labs' internship program emphasizes a remote-first, relationship-driven culture that fosters learning, collaboration, and personal development for engineering interns. The experience is shaped by a supportive environment where interns like Jessie Luk, a master's candidate in computer science, engage in meaningful projects such as working on machine learning tools and participating in cross-team hackathons. Mentorship from senior engineers like Ben Sully and the use of resources like onboarding documents and frequent communication help interns navigate the challenges of remote work, including managing different time zones. Interns are encouraged to ask questions and build connections, which enhances their understanding of production systems and the holistic integration of various technological components. The open culture at Grafana Labs allows interns to grow professionally and personally, making the program highly recommended for those seeking a comprehensive, engaging internship experience.

Open source communities thrive on active participation and collaboration, with members advancing through phases of involvement ranging from passive consumption to heavy contribution. Successful communities, like Grafana's, benefit from clear documentation, accessible forums, and transparency, which foster trust and inclusivity. The 90-9-1 rule, where a majority are passive participants, is a common dynamic in online communities, but encouraging equitable participation can help shift these ratios to more active engagement. Effective strategies include making contribution easy, celebrating positive interactions, and maintaining openness. Grafana Labs emphasizes creating a supportive environment where people can contribute at their own pace, gain expertise, and build relationships, ultimately fostering a "rising-tides-lift-all-ships" dynamic that benefits both individuals and the broader community.