November 2024 Summaries

Juraci Paixão Kröhling presents a unique "tasting menu" approach to exploring OpenTelemetry Collector configurations within Grafana Cloud, using culinary metaphors to guide users through various telemetry setup recipes. The blog post outlines four distinct configurations: an "Internal telemetry salad" for observing the Collector’s own data; a "ratatouille of events and metrics" from a Kubernetes cluster; a "bibimbap of logs" focusing on logs collection from pods; and a "Kafka buffer with two Collector layers" for handling high-volume telemetry spikes. Each configuration is designed to showcase different capabilities of the OpenTelemetry Collector, emphasizing their practical applications and potential for optimization. Complete with code snippets and a GitHub repository for detailed examples, the post aims to both educate and inspire users to experiment with and refine their telemetry data collection practices in Grafana Cloud.

Grafana Cloud offers companies a managed solution for observability, eliminating the need to manage and maintain a self-hosted open-source stack, which can be resource-intensive. Organizations like The Trade Desk, Mux, SailPoint, TeleTracking, Ultimate AI, Booking.com, and Kambi have adopted Grafana Cloud to reduce operational burdens, lower observability costs, and support modern cloud architectures, including hybrid and multi-cloud deployments. The platform's features, such as Adaptive Metrics and Adaptive Logs, help optimize metric and log volumes, thereby reducing costs and improving efficiency. Grafana Cloud also facilitates easier migration to Prometheus, offers centralized data management, and enhances security, making it a preferred choice for companies seeking scalable and robust observability solutions. These benefits have led to immediate improvements in query times, reliability, and internal satisfaction, as well as significant savings in engineering time and resources.

Migrating to a Prometheus-like ecosystem such as Grafana Cloud from platforms using different temporalities, like Datadog or Dynatrace, is streamlined by using OpenTelemetry and Grafana Alloy. This integration addresses the challenge of converting delta metrics, which report relative changes, to cumulative metrics, which represent absolute values over time. Grafana Labs has enhanced the OpenTelemetry Collector with a deltatocumulative processor to facilitate this conversion, allowing delta samples to be sent to cumulative backends such as Prometheus. However, this conversion process requires stateful operations, necessitating that each sample of a time series is consistently routed to the same collector instance. To manage this, a load-balancing exporter with streamID-based routing is employed, ensuring samples of the same series are directed to a stable collector endpoint. This setup uses Docker Compose for container orchestration, allowing metrics to be generated, routed, and converted before being sent to a backend like Grafana Cloud. The process is scalable, with even distribution of data points across multiple worker instances, but it remains in early stages, inviting community feedback and contributions to further refine its operational complexity.

Ken Buckler, an experienced information and cybersecurity expert, discusses the advantages and challenges of using microservices and Kubernetes in modern application development, emphasizing the need for effective observability solutions to manage the complexity of fragmented telemetry data. OpenTelemetry is highlighted as a vendor-neutral standard that addresses these challenges by enabling consistent and contextual telemetry data collection across various technology stacks, reducing vendor lock-in and migration costs. Grafana's suite of open-source observability tools, including Loki, Mimir, Tempo, Pyroscope, Beyla, and Alloy, enhances this by offering seamless integration with OpenTelemetry, providing a unified dashboard for comprehensive system visibility. This integration helps eliminate data silos and inconsistencies, enabling organizations to efficiently correlate and analyze diverse telemetry data, ultimately leading to improved operational efficiency and system performance. The synergy between OpenTelemetry and Grafana's tools offers a cost-effective and powerful observability solution, providing a competitive advantage in the digital age.

Grafana Loki 3.3 introduces significant enhancements to log management through the experimental use of Bloom filters for structured metadata, aiming to improve query speed and efficiency. This release marks a shift from free-text search to leveraging structured metadata, allowing users to attach metadata labels to log lines which accelerates queries by narrowing down search spaces effectively. While the feature is in public preview for large Grafana Cloud customers, it offers promising benefits for pinpointing specific log entries, especially useful for the OpenTelemetry community. Alongside these advancements, the release includes various operational improvements and bug fixes, with changes like replacing the Bloom compactor with the Bloom planner and builder components, and introducing a new build strategy for Bloom tasks. Users are encouraged to explore these features and provide feedback to enhance future developments, with comprehensive release notes available for detailed guidance.

Browser testing is crucial for modern websites to ensure optimal user experience, and the Grafana k6 browser module offers a robust solution by simulating real user interactions, collecting frontend metrics, and resolving performance issues before they affect users. Integrated with Grafana Cloud services, this module enables seamless reuse of browser testing scripts across various testing environments. Familiar to those using the Playwright framework, it provides ease of use in writing tests, while offering best practices to enhance script reliability and efficiency. Key tips include running tests locally to expedite feedback, avoiding timeouts to prevent flaky tests, considering asynchronous execution, crafting unique and stable selectors, and using thresholds or fail methods to define test pass or failure criteria. Additionally, the module supports both performance testing and synthetic monitoring, allowing for comprehensive evaluation of user journeys and backend services, while ensuring scripts are adaptable and informative even in the face of failures.

Grafana Cloud has introduced Fleet Management, a new feature in Public Preview, designed to streamline the management of observability collectors on a large scale. It provides a centralized control hub that allows users to efficiently configure, monitor, and manage hundreds or thousands of collectors across various environments and regions. With capabilities like remote configuration, targeted pipeline deployment, and comprehensive collector insights, Fleet Management helps reduce complexity, optimize costs, and improve operational efficiency. It supports as-code automation through APIs, integrates with HashiCorp Vault for secure credential management, and offers dynamic and modular pipeline configurations. Initially supporting Grafana Alloy, an open-source distribution of the OpenTelemetry Collector, the feature aims to extend support to traditional OpenTelemetry Collectors in the future. Grafana Cloud offers Fleet Management to its users, providing an easy onboarding process and encouraging feedback from early adopters.

At AWS re:Invent 2024, Grafana Labs will be actively participating and showcasing their latest offerings and innovations. The event, taking place in Las Vegas from December 2-6, will feature technical talks, labs, and keynotes, with Grafana Labs stationed at booth #2053 for attendees to meet experts, ask technical questions, and acquire swag. Highlights will include demos combining OpenTelemetry with Grafana Alloy, allowing visitors to visualize and alert on over 60 AWS offerings using the AWS Observability app in Grafana Cloud. Attendees can also explore the Grafana LGTM stack, which encompasses tools for logs, visualization, traces, and metrics, and gain insights from Grafana's annual Observability Survey. Participants have the opportunity to engage in discussions about AI/ML advancements and their role in improving observability practices, as well as learn about solutions for incident response, application observability, and performance testing.

Quest World is an innovative, text-based adventure game designed by Grafana Labs developer advocates Jay Clifford and Tom Glenn, aiming to make learning the fundamentals of observability more engaging and interactive. Utilizing OpenTelemetry and the Grafana LGTM Stack, the game challenges players to solve observability-related tasks involving metrics, logs, and traces within a fictional quest to defeat an evil wizard. The game incorporates real-world observability concepts by requiring players to use a Grafana dashboard to succeed in their quest, demonstrating the practical application of observability data types like metrics for tracking trends and logs for decision-making insights. It supports customization, allowing users to expand and adapt the adventure to their educational needs. Players can choose to play Quest World online using an online sandbox or set it up locally using open-source code available on GitHub. Additionally, the game will be showcased at AWS re:Invent 2024, offering hands-on experience and the opportunity to discover hidden Easter eggs for special rewards.

Yet Another CloudWatch Exporter (YACE), an open-source project initially started by Thomas Peitz in 2018, has become a Prometheus community project, marking a significant milestone in its development and enhancing its contribution to the Prometheus ecosystem. This transition will provide YACE with increased visibility, sustainability, and greater alignment with Prometheus best practices, benefiting from community oversight and contributions. Grafana Labs, a significant user and contributor to YACE, has played a crucial role in its growth by implementing features like improved CloudWatch metric matching to AWS resources and adopting AWS SDK v2. YACE's integration into Grafana Labs' AWS Observability app underscores its importance due to its ability to discover AWS resources and their related CloudWatch metrics, with future plans from Grafana Labs to continue seamless contributions and assist in migrating from cloudwatch_exporter to YACE. The project’s transition to prometheus-community signifies a promising future for YACE in the AWS observability realm.

Grafana 11 introduced several key advancements aimed at enhancing user experience, scalability, and extensibility, as discussed in a recent episode of the "Grafana's Big Tent" podcast featuring Grafana Labs leaders. The release focused on simplifying data exploration, visualization, and alerting, with features like Explore Metrics, which aids both novice and experienced users by automating complex tasks. Subfolders, a highly requested feature, and improvements to data visualizations further enhance user interface management. The transition from Angular to React, a lengthy but necessary overhaul, has been completed, allowing for more dynamic dashboard features. Grafana Labs also outlined how community feedback, hackathons, and direct customer input contribute to their development process. Looking ahead, Grafana 12 is set to explore new dashboard architectures and expand data source integrations, which have already seen rapid growth with numerous new additions this year. The podcast episode also highlighted the company's focus on ensuring backward compatibility and reliability through Grafana Cloud's frequent updates, thus maintaining high software quality.

Grafana Beyla, an open-source eBPF auto-instrumentation tool by Grafana Labs, has significantly advanced since its launch, fostering a robust community of external contributors who have shaped its development. Beyla excels in providing easy access to application and service graph metrics, supporting a range of protocols beyond its initial HTTP and gRPC, and is widely used for OpenTelemetry observability. Despite challenges with distributed traces due to eBPF's complexities, the team has made strides toward enabling full support in the upcoming 2.0 release. Production deployment required optimizing resource utilization and managing eBPF permissions to address security concerns. Future plans include leveraging the OpenTelemetry eBPF Profiler for deeper insights and enhancing language support for top programming languages like Java, .NET, and Python, without compromising Beyla's language-agnostic capabilities. The project’s evolution reflects the collaborative efforts within its community and the ongoing pursuit of improved observability solutions.

Grafana Cloud, a fully managed observability platform powered by the Grafana LGTM Stack, has introduced several updates to enhance user experience, including redesigned dashboard filters that streamline the creation process and support for multi-select operators. Bar gauge visualizations now feature legend support to improve standardization across panels, while table visualizations benefit from an improved inspect value experience with new tabs for data inspection. The platform also enables correlations to external URLs, allowing users to link data sources directly to external platforms such as Salesforce. Furthermore, role-based access control (RBAC) has been extended to plugins and Grafana Alerting, allowing administrators to manage permissions for plugins, contact points, mute timings, and notification templates. Grafana Cloud also announced release freezes during the holiday season to prevent scheduled updates, though critical changes may still occur. Users can explore these features with Grafana Cloud's free tier.

Grafana recently addressed a medium severity security vulnerability, CVE-2024-9476, found in the Grafana Cloud Migration Assistant, a feature available in public preview from version 11.2.0. This vulnerability allowed unauthorized access to resources across different organizations within the same Grafana instance, affecting users utilizing the Organizations feature. Grafana has released patched versions 11.3.0+security-01 and 11.2.3+security-01 to rectify the issue, urging users to upgrade promptly. The vulnerability was discovered during internal testing, and immediate steps were taken to disable the problematic feature, which has since been re-enabled following the application of security patches. Grafana Labs encourages users to report security issues through their dedicated page and provides detailed timelines and post-incident reviews to ensure transparency.

Martin Falch, an expert in CAN bus data and co-owner of CSS Electronics, has developed a solution to enhance real-time visualization of CAN-to-USB data using Grafana dashboards and the MQTT plugin. This approach addresses the limitations of traditional software tools, which often struggle to provide customizable real-time data visualization. By integrating CSS Electronics' CANedge/CANmod devices with Grafana, users can set up custom dashboards quickly, utilizing the MQTT data source plugin and a Python API for real-time data streaming and decoding. This enables engineers across industries such as automotive and industrial equipment manufacturing to diagnose issues and monitor performance effectively. The setup can be scaled to handle multiple devices and data sources, offering flexibility in data management and remote visualization capabilities over local networks or the internet. Additionally, the solution supports dynamic signal selection via Grafana variables and separate dashboards for specific analyses, enabling efficient data management and real-time alerts for certain events, demonstrating its potential for diverse applications in telematics, predictive maintenance, and more.

Grafana has experienced a significant year, highlighted by the April release of Grafana 11.0 at GrafanaCon 2024, which introduced a queryless experience and custom visualizations. Subsequent minor releases have enhanced data sources and visualizations, culminating in the 11.3 release with Scenes-powered dashboards. As the year ends, Grafana is implementing release freezes during the holiday season, affecting various products but allowing for essential updates in case of operational or security incidents. A new naming convention for security releases now appends "+security" to versions, clarifying the nature of updates and improving understanding of their urgency. Release information has been integrated into the Grafana documentation under "Upgrade Strategies," ensuring better access to details about Grafana Enterprise and OSS releases. Grafana Cloud remains a user-friendly platform for handling metrics, logs, traces, and dashboards, offering a free tier and scalable plans.

Grafana dashboards offer a versatile platform for visualizing and correlating data from various sources, enabling users to troubleshoot efficiently, make informed decisions, and optimize monitoring processes. These dashboards are composed of panels that transform raw data into visualizations using data source plugins, queries, and optional transformations. Users can construct complex visualizations by employing multiple queries from the same source, joining disparate data frames through transformations, and executing mixed-source queries for different data formats and databases. Grafana supports over 150 data source plugins, allowing for a wide range of integrations, while its open-source nature and flexible transformations enable database-like operations within the platform. To enhance performance when dealing with large data volumes, it's advisable to filter data at the database layer before importing it into Grafana. For those seeking to explore more about Grafana's capabilities, comprehensive documentation and community-submitted dashboards offer valuable resources, and Grafana Cloud provides an accessible way to begin using metrics, logs, traces, and dashboards.

Prometheus 3.0 has introduced enhancements to facilitate better integration with OpenTelemetry, addressing challenges such as resource attribute handling, UTF-8 support, and differing data collection models. Prometheus users can now promote OpenTelemetry resource attributes to metric labels, which helps in maintaining data clarity without overwhelming the system's performance. The new PromQL function, info, simplifies querying by mitigating conflicts from changing resource attributes. Additionally, Prometheus 3.0 supports the OpenTelemetry protocol (OTLP) endpoint for stable data ingestion, offering improvements in translation efficiency and resource management. The release also includes UTF-8 support, aligning Prometheus with OpenTelemetry standards and accommodating diverse character inputs. While delta temporality is not yet fully supported, efforts are underway to enable this feature, reflecting a broader commitment to making Prometheus a robust backend for OpenTelemetry metrics. The blog encourages community participation in further development and provides resources for users seeking more information or wishing to contribute to the ongoing improvements.

Grafana Labs is participating in KubeCon + CloudNativeCon North America 2024 as a silver sponsor, where they will showcase advancements in observability and monitoring technologies. The event, taking place in Salt Lake City, Utah, will feature various talks by Grafana Labs team members on topics such as managing observability data volumes, using eBPF for system observation, and cost-saving measures in Prometheus Remote Write 2.0. Additionally, Grafana Labs will present updates on OpenTelemetry's expanded capabilities and share insights into maintaining continuous reliability in their systems. The conference will also highlight the release of Prometheus 3.0, offering new features and a refreshed user experience, with opportunities for attendees to engage with maintainers and learn how to leverage these updates effectively.

In a blog post by Vasil Kaftandzhiev from Grafana Labs, the author explores the significance of a newly released alert system feature in the Kubernetes Monitoring app on Grafana Cloud. Initially overlooked as a minor addition, this feature allows users to create alerts directly from panels by setting thresholds on queries, making incident management more efficient. During a live demo for a cost-conscious customer, the potential of this feature became apparent as it enabled the setup of automated alerts for cost control and resource management, sparking excitement among stakeholders. The experience highlighted the importance of utilizing seemingly small features that can have a profound impact on operations and cost savings. The author encourages users to explore hidden features, emphasizing that even minor tools can transform into powerful assets for managing Kubernetes infrastructure.

In a significant expansion move, Grafana Labs has launched a new Grafana Cloud region in Indonesia, leveraging the AWS Asia Pacific (Jakarta) Region to provide enhanced connectivity and compliance with local data regulations. This development, made possible through collaboration with local partner iZeno, aims to meet the rising demand for data sovereignty, improved performance, and adherence to Indonesian data laws. The initiative underscores Grafana's commitment to the Asia-Pacific region, following recent expansions in Japan, Australia, India, and Singapore. This expansion is designed to help organizations in Southeast Asia fully utilize Grafana Cloud's capabilities, offering faster and more reliable services tailored to the unique needs of the market. The launch will be celebrated with an event on November 12, featuring insights from industry leaders and providing opportunities for networking and learning about observability trends and best practices.

In a recent episode of "Grafana's Big Tent" podcast, the discussion centered around eBPF, an innovative technology that allows for dynamic program loading into the Linux kernel, enhancing observability and performance without compromising system stability. Hosted by Grafana Labs' Tom Wilkie and Matt Toback, along with guests like Liz Rice from Isovalent, the conversation explored eBPF's potential in improving application observability through tools like Grafana Beyla, an auto-instrumentation tool that provides insight into various programming languages without needing to modify applications. The podcast also touched on eBPF's security implications, highlighting its ability to filter and manage kernel events efficiently, such as with Cilium's Tetragon project. The episode concluded with discussions on future developments for eBPF, emphasizing its growing role in networking and security, and its integration into products like Cisco's Hyper Shield, showcasing the ongoing evolution and adoption of eBPF across different domains.