June 2024 Summaries

A security advisory has been issued concerning Grafana Loki, an open-source log aggregation tool, due to unauthorized data write attempts to Amazon S3 buckets caused by its default configuration settings. This issue was initially highlighted in a blog post by Maciej Pocwierz, who incurred a $1,300 charge after unauthorized write attempts to his S3 bucket, named using Loki's default settings. The problem arose when the default bucket names in the Loki Helm chart created unintended write targets for AWS users. In response, Grafana Labs has updated the Loki Helm chart to prevent the use of default S3 bucket names, except when using MinIO, and has coordinated with AWS and Google Cloud to ensure that no unauthorized data collection occurs. While AWS has amended its billing policy to no longer charge for unauthorized requests, Google Cloud users remain unaffected, and Microsoft Azure users are inherently protected due to its bucket namespace limitations. The security update advises users to upgrade their Loki Helm charts to patched versions and modify bucket names to avoid potential misconfigurations.

ObservabilityCON 2024 is set to take place on September 24-25 in New York City, focusing on the latest advancements in the Grafana LGTM Stack and offering insights from industry experts. This flagship event will feature a comprehensive agenda, including keynote speeches, technical sessions, community insights, and hands-on workshops to enhance participants' skills in observability practices. Attendees will have the opportunity to learn about monitoring modern applications, OpenTelemetry, and Grafana Cloud Logs, with early bird tickets offering a 50% discount. The event aims to provide actionable strategies for improving organizational observability and customer satisfaction, while also offering networking opportunities with professionals in the field. Participants are encouraged to register promptly due to limited discounted tickets and to consider attending workshops, which require separate tickets alongside a full conference entry.

Grafana 11.1 introduces several enhancements to improve user experience, including new visualization features, updates to Grafana Alerting, and increased accessibility. Users can now enjoy text wrapping in table panels and customizable color options for percent changes in stat panels, as well as the availability of XY charts for visualizing relationships between variables. The updated Grafana Alerting includes a redesigned settings page for better management of alert instances and new features like alert templates and OAuth2 authentication for Alertmanager or Mimir. Accessibility improvements include keyboard support for geomaps and panel shortcuts, enhanced heading structures for screen readers, and reduced motion settings to accommodate users sensitive to animations. Additionally, the update emphasizes community engagement and offers a free tier for Grafana Cloud to help users transition smoothly to the new version.

Grafana Cloud has introduced several updates aimed at enhancing user experience and functionality in its fully managed observability platform. These updates include new data visualization features, such as customizable percent change colors and text wrapping within table cells, alongside the general availability of XY charts for improved data representation. The platform also enhances performance test analysis through Cloud Insights, which streamlines root cause identification by analyzing telemetry data. Additionally, Grafana Explore now supports log filtering and pinning, facilitating easier navigation and analysis of logs, while private source map uploads in Frontend Observability protect user IP by enabling de-minified stack traces. Grafana Alerting has also been improved with rule-specific silences and a template selector for alert notification messages, and a new Catchpoint integration offers real-time insights into digital performance metrics. These updates come with a generous free tier, encouraging users to explore and utilize the platform's capabilities.

The text explores how modern observability tools like OpenTelemetry, Grafana Cloud, and Digma.ai are transforming the way developers approach software development by integrating observability into their daily workflows. Traditionally, developers depended on logs for troubleshooting, while metrics and traces were handled by IT teams, leading to a reactive approach to problem-solving. The narrative highlights the use of OpenTelemetry for simplified Java application tracing and Grafana's visualization capabilities, which together make observability more accessible to developers. It emphasizes the importance of shortening the feedback loop to quickly detect and address issues by connecting Java applications to Grafana Cloud and Digma.ai, which can analyze traces directly in the IDE. The process includes setting up a Grafana Alloy container to manage observability data and using the Digma plugin to integrate observability insights into the development environment, thereby promoting a proactive and continuous feedback approach to software development.

The U.S. Department of Defense (DoD) is increasingly adopting open-source software (OSS) and DevSecOps principles to modernize its software development processes, prioritizing speed, quality, and security. This shift is part of a broader strategy to create an ecosystem of software factories, utilizing tools like Platform One's Big Bang, which provides infrastructure provisioning, CI/CD pipelines, and security scanning. Big Bang 2.0 has integrated open-source projects from Grafana Labs, such as Grafana Loki, Grafana, and Grafana Tempo, to enhance observability with lower resource costs and no license requirements. Grafana offers a centralized platform for monitoring and improving software reliability by consolidating data from various sources, enabling DoD software factories to proactively manage their systems with role-based dashboards and comprehensive insights into key performance indicators.

Grafana Labs has released new open-source code for an OpenTelemetry Datadog receiver, enabling the translation of Datadog metrics into the OpenTelemetry Protocol (OTLP) format, which can be integrated with systems like Prometheus, Grafana Mimir, or others. This initiative aligns with Grafana's commitment to open-source values and interoperability, reducing vendor lock-in and allowing users to choose their observability strategies. The translation process for Datadog metrics, such as Count, Gauge, and Rate, is straightforward, while the Distribution metric type requires more complex mapping to OpenTelemetry's Exponential Histogram metric type. The feature is currently experimental, and Grafana Labs is working with the OpenTelemetry community to merge the code into the upstream main branch. Users can utilize Grafana Alloy v1.2.0 to test the receiver and processor, with guidance provided for configuring the Datadog Agent to forward metrics to Alloy. Grafana also offers support for migrating users from Datadog to Grafana Cloud, highlighting its comprehensive and user-friendly platform for metrics, logs, and traces.

OpenTelemetry provides a framework for collecting telemetry data from various resources, such as hosts and services, to help identify issues like increased application latency. By associating resources with attributes like host ID and service instance ID, OpenTelemetry allows users to filter and analyze data to pinpoint performance issues. Resource detectors, which are available for different platforms like AWS, Azure, and Google Cloud, gather this information, and the insights can be visualized using tools such as Grafana Cloud Application Observability. This process enables users to conduct root cause analysis by examining metrics and traces, ultimately allowing for informed decisions to improve application performance. Grafana Labs is actively enhancing resource detectors within different OpenTelemetry SDKs to support a wide range of languages and platforms, emphasizing the vendor-neutral nature of OpenTelemetry and its applicability to various backend systems.

Dell Technologies successfully migrated from SolarWinds to Grafana Cloud to consolidate its observability tools, achieving a more integrated and efficient system without losing functionality. The transition, led by Brian Murphy from Dell's Infrastructure Solutions Group, enabled Dell to access previously unavailable data and reduce unnecessary metrics using Grafana's Adaptive Metrics feature, which resulted in cost savings and streamlined operations. The new platform improved alert management, facilitated problem-solving for 13,000 engineers, and provided executives with insightful dashboards, making Grafana Cloud the primary tool for incident management. Dell plans to continue expanding its use of Grafana Cloud by onboarding more teams and services, with an emphasis on smart alerts and experience-focused monitoring, aiming to enhance the overall observability across its infrastructure.

Grafana Cloud has introduced several new features in its Application Observability tool to enhance anomaly detection and root cause analysis, addressing customer needs for better understanding and contextualized analytics of observability data. The tool, which supports OpenTelemetry and Prometheus, offers developers and SREs the ability to unify application and infrastructure insights. Key features include time frame comparison to identify anomalies over different periods, automatic baselining to account for seasonal variations, and tools like group-by and filter-by to narrow down problem dimensions based on specific attributes. Additionally, in-context navigation with Grafana data links allows users to quickly transition from metrics to traces and logs, streamlining root cause analysis. Despite a growing emphasis on application observability, only 41% of organizations are actively incorporating it, though 53% consider it more relevant than the previous year. Grafana Cloud offers a free tier for those interested in exploring these capabilities.

Grafana is transitioning from using API keys to service account tokens (SATs) for enhanced security and flexible access control, with all existing API keys continuing to function seamlessly until their migration to SATs by January 2025. This change, which includes the removal of the api_key resource from the Grafana Terraform provider in version 3, aims to improve security by reducing the surface area and operational costs associated with supporting both authentication methods. Service accounts, available since Grafana 9.1, simulate Grafana users to run automated workloads and offer more granular, role-based access controls compared to API keys. Developers are encouraged to transition their automation scripts and tools, such as Terraform, to use SATs by the end of 2024, with a detailed migration path provided. Grafana 11, introduced at GrafanaCON 2024, includes new features to securely scale within organizations, with detailed sessions available on demand. Grafana Cloud offers a free tier for easy access to metrics, logs, traces, and dashboards.

Expanding their partnership, Grafana and Embrace have introduced a new integration that enhances mobile app observability by leveraging open standards like OpenTelemetry. This collaboration enables users to visualize and analyze real-time mobile metrics through Grafana dashboards, providing a comprehensive view of user experiences from mobile devices to cloud infrastructure. Embrace's solution focuses on user-centric mobile app observability, offering transparency and enterprise-supported insights through its OpenTelemetry SDKs, which facilitate the connection of mobile telemetry to backend performance data. The integration allows teams to correlate mobile data with other observability data in Grafana Cloud, improving their ability to address performance issues proactively. This development is aimed at modernizing observability practices and delivering superior user experiences, while Grafana Cloud offers a user-friendly platform with a range of plans and a generous free tier.

In a detailed guide, Michael Derynck outlines the steps to configure an incident response system using Grafana OnCall, paired with Twilio, to route alerts via SMS and voice calls. The process begins with setting up a basic integration where a phone number in Twilio sends alerts to a webhook in Grafana OnCall. The guide then covers the creation of escalation chains and routes in Grafana OnCall to handle different alert types and ensure notifications reach the appropriate personnel. Twilio's Studio flows are utilized to manage incoming SMS and calls, converting them into alerts sent to Grafana OnCall. The guide also includes instructions for testing the setup and troubleshooting common issues, as well as expanding the configuration to route alerts based on specific criteria. This setup allows for scalable and customizable incident management, ensuring alerts are efficiently communicated to on-call teams.

Roblox, a massively popular platform with over 71.5 million daily users and 2.4 million developers, has revolutionized its runtime diagnostics process to ensure seamless user experiences by integrating Grafana Pyroscope for on-demand profiling. Previously, debugging was a manual, inefficient, and security-laden process requiring engineers to access production environments to diagnose issues. By embedding the Pyroscope Profiler into their container-based images and leveraging Grafana Cloud Profiles, Roblox engineers can dynamically manage profiling, leading to faster identification and resolution of bottlenecks, such as optimizing poor-performing libraries and improving CPU usage. This shift has led to significant performance gains, including a 36% reduction in CPU usage from replacing a JSON library and a 10x improvement in throughput for a new service. Roblox aims to further enhance this system with event-based profiling and ultimately continuous profiling, marking a pivotal step toward integrating runtime diagnostics as a critical pillar of observability alongside logs, metrics, and traces.

Grafana plugins are essential for customizing and extending the capabilities of the Grafana platform, and the blog post provides six key tips to help developers improve their plugins before publication. These tips include building a comprehensive README file to facilitate user understanding and using metadata to enhance discoverability in the plugin catalog. Developers are advised to specify compatible Grafana versions, utilize provisioning for testing environments, and validate their plugins using the plugin validator tool to ensure compliance with security and structural guidelines. Additionally, using GitHub Actions can automate the release process, helping to catch errors early and streamline the publication workflow. The post emphasizes the vibrant community-driven ecosystem of Grafana plugins and encourages developers to engage with community resources for further guidance and support.

Grafana Alloy clustering is an advanced OpenTelemetry Collector distribution that enables efficient scraping of nearly 20 million Prometheus metrics through its powerful agentless exporters, which work without requiring users to run applications on their infrastructure. This system, which operates on Grafana Cloud, leverages clustering capabilities to distribute Prometheus scrape jobs across a network of Alloy nodes using a gossip-based protocol, enhancing scalability and high availability. The agentless exporters, specifically designed for Grafana Cloud, facilitate data retrieval from providers like Amazon CloudWatch without local infrastructure setup. The migration from the previous Grafana Agent scraping service to Alloy clustering solved previous complexities related to dual write problems and extra persistence layers by dynamically pulling configuration from remote sources. The transition process involved deploying new Alloy nodes, configuring them with import.http blocks, and incrementally migrating scrape jobs while maintaining resource efficiency comparable to the former system. Overall, Grafana Alloy clustering simplifies and optimizes the scraping of metrics and offers substantial improvements over its predecessor, making it a robust solution for managing large-scale metric ingestion in Grafana Cloud.

Grafana is a versatile open-source platform used for monitoring and visualizing data, with various transformation functions allowing users to manipulate and present data effectively before visualization. The author, an experienced Grafana user and engineering director at Grafana Labs, highlights five key transformations: "Group by" for creating bar charts, "Organize fields by name" for renaming, hiding, and reordering columns, "Filter data by value" for refining data with conditions, "Sort by" for arranging data in a desired order, and "Partition by values" for splitting datasets into multiple series. Additionally, Grafana now includes a feature for calculating any percentile, enhancing data analysis capabilities. The platform supports over 30 transformation functions, offering expansive customization for data visualization and analysis. Grafana Cloud provides an accessible entry point with a free tier for users to explore its features.

Grafana Tempo 2.5 introduces several updates including performance improvements, TraceQL metrics enhancements, and support for new streaming endpoints, alongside a shift to the vParquet4 format. TraceQL metrics, still experimental, now allow users to create dynamic metrics from traces with new methods such as quantile_over_time() and histogram_over_time(), enabling aggregation of numerical values. The update includes a major refactor of the query frontend to support streaming gRPC endpoints, aiming to expand beyond the currently supported TraceQL streaming search endpoint. The introduction of vParquet4, although not yet default, prepares for future TraceQL features with added support for querying events, links, and arrays. Tempo 2.5 also brings breaking changes like the removal of the original vParquet format and a security update running the process as a non-root user, requiring attention during upgrades. Looking ahead, Grafana Labs focuses on making TraceQL metrics generally available and extending the TraceQL language, while also considering architectural changes to improve performance and cost-efficiency. For community engagement, users are encouraged to participate in discussions, join community calls, and explore open positions at Grafana Labs, with Grafana Cloud offering a free tier to get started with Tempo.