November 2023 Summaries

The 2024 Grafana Labs Observability Survey reveals key insights into the current state and trends in observability practices among organizations. A significant majority of respondents are using multiple observability technologies, with many incorporating open-source solutions like Prometheus and OpenTelemetry, indicating a rise in OSS adoption. There is a growing trend towards centralizing observability, which 80% of respondents report has resulted in time and cost savings, notably through improvements in mean time to repair (MTTR) and reduced overheads. However, only a small percentage of organizations, 16%, have achieved a fully systematic approach, while proactive strategies are more common than reactive ones. The survey also notes increasing interest in newer practices such as Service Level Objectives (SLOs) and eBPF, highlighting the expanding scope of observability in addressing application performance, microservices, and supply chain security. As the market matures, the findings suggest a shift towards more integrated and comprehensive observability frameworks, although the final results are still pending further community input.

Grafana Agent v0.38 introduces over 50 updates aimed at enhancing user experience for SREs and developers, featuring new OpenTelemetry components for improved data filtering and telemetry reception, along with the ability to profile Python applications using the pyroscope.ebpf component. It also includes configuration improvements in Grafana Agent Flow mode, supporting Go-style raw strings for easier regex handling and returning null for non-existing keys, enhancing robustness in pipeline building. Log collection is improved with better detection of rolled log files and support for clustering in Kubernetes, as well as enhancements to the Write-Ahead Log implementation for improved data handling in case of system interruptions. The release also focuses on scalability by enhancing the resilience of large-scale deployments, ensuring that slow components do not impact other running pipelines, and invites users to explore more features and updates through Grafana's documentation and community platforms.

InfluxDB, an open-source time series database, paired with Grafana, a visualization tool, can be utilized to analyze and display data trends over time, such as CPU load differences. The blog post details a step-by-step process for using InfluxDB's built-in difference() function to calculate the change in a value over a specified time duration, exemplified by a one-day difference in CPU load data. It explains how to construct an InfluxQL query to return the desired data, emphasizing the use of the difference() and last() functions within a SELECT clause, and the necessity of a GROUP BY interval to organize results by time. The guide further instructs on integrating this query within Grafana to visualize the data, highlighting the platform's versatility in handling various data sources. For those interested in exploring further, the post suggests referring to technical documentation and Grafana Cloud's offerings, including a free tier, for expanding one's data analytics capabilities.

AWS re:Invent 2023, a major cloud computing conference held in Las Vegas from November 27 to December 1, is hosting numerous technical sessions, labs, and keynotes, where Grafana Labs is participating as a sponsor. Attendees can visit Grafana Labs at booth #1088 in the Venetian Expo Hall to experience live demonstrations, meet with experts, and explore the Grafana LGTM Stack, which includes Loki for logs, Grafana for visualization, Tempo for traces, and Mimir for metrics, to enhance their observability strategies. The event will feature prominent Grafana Labs executives, including CEO Raj Dutt and CTO Tom Wilkie, who will be available for discussions throughout the week. Additionally, Grafana Labs will showcase how Grafana Cloud can integrate with over 60 AWS services to provide a centralized view of critical observability data, highlighting capabilities for AWS monitoring. Opportunities to meet with Grafana Labs can also be arranged in advance.

Grafana has introduced a new "No basic role" feature that enhances user management and security by eliminating default permissions for users and service accounts, addressing issues from previous role-based access control systems where the default Viewer role could unintentionally grant excessive permissions. This update, available in Grafana 10.2 for both Grafana Cloud instances and Enterprise binaries, allows administrators to more precisely control access by starting with a permissionless role and assigning necessary permissions through Team Role sync or RBAC (Role-Based Access Control). This approach not only simplifies access management but also minimizes security risks by reducing unnecessary access to sensitive resources, aligning with the principle of least privilege. The feature, developed in response to community feedback, is part of Grafana’s ongoing efforts to improve security and user management, and users are encouraged to provide feedback to help further refine the tool.

OpenTelemetry Collectors are essential tools for managing telemetry data, acting as intermediaries to gather, filter, and transmit logs, metrics, and traces from applications to backends like Grafana. While it's possible to send data directly to backends, using a collector offers significant advantages, such as filtering sensitive information, managing data costs, and decoupling services from specific backend endpoints. Collectors enable organizations to redact sensitive data before transmission and control the volume of data sent, which is crucial for cost management given Grafana's data ingestion fees. Additionally, collectors provide operational flexibility by allowing services to send telemetry to a single collection point, simplifying maintenance and backend changes without altering individual application configurations. Although direct transmission may work for simple or early-stage projects, collectors are recommended for complex, production-level systems to ensure efficient data management and avoid potential pitfalls such as privacy issues and service downtime during backend maintenance.

CI/CD observability goes beyond simple monitoring by providing a comprehensive understanding of the entire software delivery pipeline, from code check-ins to deployments. At Grafana Labs, efforts to enhance CI/CD observability have culminated in the development of GraCIe, an opinionated application plugin designed to simplify the analysis of CI/CD systems for users by leveraging Grafana's existing telemetry tools like Grafana Tempo, Grafana Loki, and Prometheus. This initiative aims to address common challenges such as flaky tests, performance regressions, and misconfigurations by providing visibility and insights into CI/CD processes, ultimately reducing deployment fears and promoting continuous improvement. The project envisions a future where CI/CD observability is standardized across platforms, empowering teams to make informed decisions and fostering a culture of transparency and accountability.

ObservabilityCON 2023, held in London, showcased advancements in open-source observability, emphasizing strategies to scale and modernize using the LGTM Stack, which includes Loki for logs, Grafana for visualization, Tempo for traces, and Mimir for metrics. Sessions, now available on demand, highlighted the integration of AI and machine learning into observability practices, offering insights into user experiences and system performance. Key sessions featured discussions on scalability, performance, and the observability journeys of organizations like Pipedrive, Wise Payments, and Maersk, who shared their migration experiences to the LGTM Stack. The event also covered Grafana Labs' developments, including the release of Grafana Beyla 1.0 and the acquisition of Asserts.ai, underscoring the evolving role of AI/ML in observability.

Grafana Labs has introduced its OpenTelemetry Distribution for .NET, designed to enhance application observability with minimal setup and configuration. This tailored version of OpenTelemetry components is optimized for use with Grafana Cloud Application Observability, allowing users to easily emit telemetry data while reducing observability costs. The distribution includes bundled instrumentations and resource attributes necessary for effective monitoring, and it is tested for stability with Grafana Cloud. Users can start quickly by sending telemetry directly to Grafana Cloud, though using a Grafana Agent or OpenTelemetry Collector is recommended for production. Grafana remains committed to the upstream OpenTelemetry project, ensuring compatibility and contributing new features for the broader community. Future plans include integrating auto-instrumentation and supporting upcoming .NET releases to provide a seamless, zero-code-change experience for users.

The Grafana OpenTelemetry Distribution for Java is introduced as a customized version of OpenTelemetry components, designed to enhance application observability with minimal setup and configuration. Optimized for Grafana Cloud Application Observability, this distribution supports any OpenTelemetry-compatible databases, allowing users to efficiently manage telemetry data and reduce observability costs. The distribution is tested for stability and integrates seamlessly with Grafana Cloud, bypassing the need for additional setup like Grafana Agent or OpenTelemetry Collector for initial configurations. Users are encouraged to engage with the Grafana community for support and feedback, and the distribution is designed to be compatible with other OpenTelemetry components, enabling easy migration to upstream OpenTelemetry if desired. Grafana emphasizes its commitment to contributing to the OpenTelemetry project, ensuring that its distributions remain aligned with upstream developments and are compatible with various databases and vendors.

Grafana Labs has introduced Grafana SLO, a feature within Grafana Cloud, aimed at simplifying the creation, management, and scaling of service level objectives (SLOs) to enhance service reliability and performance measurement. Initially developed to address frequent false alerts from an SLA that monitored request latency, Grafana SLO provides users with a guided UI, prebuilt dashboards, and customizable alerting rules, reducing alert fatigue and enhancing data-driven decision-making. This tool enables organizations to set and manage SLOs efficiently, offering features like Terraform integration and an SLO API for scalability as code, thus aligning development and operations teams towards common service expectations. By offering executive-level visibility into service health, Grafana SLO supports strategic decision-making and resource allocation, asserting its importance as highlighted in Grafana Labs' 2023 Observability Survey, where a significant number of respondents indicated their use or interest in SLOs.

Grafana Cloud has introduced a new cost management hub aimed at helping organizations manage and optimize their observability expenses, particularly as many face rising costs with increased scale. Key features include the Log Volume Explorer, now in public preview, which helps users investigate sources of high log volumes by allowing a detailed breakdown of log data by labels, and Adaptive Metrics, a generally available tool that reduces unnecessary metrics by aggregating them into lower cardinality versions. The Usage Attribution Report, also in public preview, offers a granular analysis of monthly metrics usage by teams, facilitating practices such as chargeback and showback for cost accountability. This suite of tools is designed to give Grafana Cloud users greater control over their metrics and logging costs, with features accessible across different subscription tiers, including free and paid options. These new capabilities were unveiled at ObservabilityCON 2023, along with demonstrations and documentation to support users in leveraging these tools effectively.

ObservabilityCON 2023, held in London, showcased significant developments from Grafana Labs aimed at enhancing the open-source observability community's capabilities. Key announcements included the acquisition of Asserts.ai, which will aid Grafana Cloud users in more effectively understanding their observability data, and the general availability of Application Observability, designed to expedite root cause analysis and reduce mean time to resolution. The event also highlighted the release of Grafana Beyla 1.0, an eBPF-based application auto-instrumentation project, which simplifies the observability process by allowing users to instrument services without modifying source code. Additionally, Grafana Labs introduced Grafana SLO for easier management of service level objectives and a new cost management hub in Grafana Cloud to optimize expenses. The company is also exploring AI/ML capabilities, with features like Sift, an automatic diagnostic assistant, and generative AI tools for enhanced dashboard creation and query writing, all contributing to a more integrated and efficient observability experience.

Grafana Beyla 1.0 is an open-source eBPF auto-instrumentation tool designed for application observability, offering zero-code instrumentation for OpenTelemetry and Prometheus applications. Released at ObservabilityCON 2023, Beyla 1.0 introduces multi-process support, allowing a single instance to monitor various applications across different programming languages, enhancing resource efficiency, especially in microservice environments. The release also includes a heuristic HTTP path decorator to manage high cardinality in monitoring metrics and built-in support for Grafana Cloud, simplifying telemetry data integration. While Beyla currently supports single span traces, plans are underway to incorporate distributed traces, beginning with a prototype for Go applications, aiming to expand across other languages. Beyla's capabilities, including its compatibility with Grafana Tempo and the OpenTelemetry Collector, make it a versatile tool for developers seeking to enhance application visibility without modifying underlying code.

Grafana Labs has introduced Application Observability within Grafana Cloud, enhancing its existing LGTM Stack (Loki, Grafana, Tempo, Mimir) to provide an out-of-the-box solution for application monitoring. This new capability aims to reduce mean time to resolution (MTTR) and improve application reliability by offering preconfigured dashboards, workflows, and alert setups, enabling users to detect anomalies and identify root causes quickly. With native support for OpenTelemetry and Prometheus, Application Observability allows users to utilize OpenTelemetry auto-instrumentation and PromQL-based query languages, ensuring flexibility and avoiding vendor lock-in. The platform facilitates seamless correlation of metrics, logs, and traces across different layers of applications, using tools like the Service Map view and distributed traces to pinpoint issues effectively. An example of its application is demonstrated through the OpenTelemetry Community Demo, highlighting its ability to troubleshoot and resolve elevated error rates in microservices. The feature is available to all Grafana Cloud users, including those on the free tier, and is designed to integrate easily with existing telemetry systems.

Grafana Labs has acquired Asserts.ai, a startup focused on simplifying the exploration of telemetry data and enhancing root cause analysis, to address customer demands for easier understanding of observability data and deeper analytics. This acquisition, announced at ObservabilityCON in London, includes integrating Asserts into Grafana Cloud to provide a more seamless and contextual user experience. Asserts offers a contextual layer for Prometheus metrics, facilitating insights into system component relationships and improving alert troubleshooting and data querying. The integration aims to streamline operations by offering pre-made alerting rules and dashboards, and it is set to become an integral part of the LGTM Stack (Loki, Grafana, Tempo, Mimir). Asserts, founded by Manoj Acharya in 2020, has developed the SAAFE Model, a metric meta model for open observability data. The collaboration with Grafana Labs is expected to enhance the visualization and understanding of system relationships, pushing forward the unification of infrastructure and application observability within Grafana Cloud.

ObservabilityCON 2023, hosted by Grafana Labs, is set to take place on 14-15 November in London, featuring a keynote from key Grafana figures including CEO Raj Dutt and CTO Tom Wilkie, along with insights from the engineering team. The event will explore current trends in open source observability, with presentations on the Grafana LGTM Stack and discussions from experts at companies like ComplyAdvantage and Maersk. Attendees can participate in breakout sessions on topics such as unifying application and infrastructure observability, building observability at scale, and managing metrics costs with Grafana Cloud. Networking opportunities will be available, and the event will offer both in-person and virtual participation, with livestreams and on-demand sessions accessible via Grafana Labs' Community Slack and social media platforms.

Grafana Labs transitioned from using Cluster Autoscaler (CA) to Karpenter for managing its Amazon Elastic Kubernetes Service (EKS) clusters to address inefficiencies and reduce both costs and complexities. Initially, CA was chosen for its simplicity and proven track record, but it soon became evident that it was not optimal for Grafana's needs due to issues like underutilized nodes, limited flexibility with instance types, and difficulties with using AWS Spot Instances. Grafana Labs found Karpenter, an open-source solution primarily developed by AWS, to be more aligned with their requirements because of its just-in-time node provisioning, native support for capacity type prioritization, and its ability to directly interact with AWS EC2 API for optimal instance selection. This switch resulted in a significant reduction in idle capacity and costs, improved reliability, and better developer experience without disrupting service to users. Although Karpenter is still under active development and currently limited to AWS, the benefits it brought to Grafana Labs, such as enhanced disaster recovery capabilities and simplified infrastructure management, outweighed the trade-offs of adopting a cloud-specific scaling solution.

Grafana's recent redesign of panel headers, introduced in the 9.5 release, marks a significant shift from center to left-aligned titles, a decision driven by usability and accessibility challenges identified over time. This change aims to address issues like non-intuitive menu access and overlapping panel elements, which hindered user experience, particularly for those using complex panel configurations. The new design allocates distinct spaces for each element, aligns with the intuitive UI pattern of having the title on the left and the menu on the right, and accommodates long titles and additional elements, enhancing scalability for various use cases. Despite facing technical challenges, such as outdated components and ensuring a consistent look across Grafana's products, the redesign has been implemented with flexibility in mind, supporting plugin developers and projects like Grafana Scenes. While initial community responses have been mixed, Grafana's team remains committed to refining the panel header design based on user feedback, continuing their tradition of community engagement and iterative improvement since the platform's inception.

Grafana Labs has launched the Saga Design System to provide a cohesive visual language across its products, enhancing the user experience for its LGTM Stack. This system offers a centralized resource for design elements, patterns, and guidelines, promoting consistency and ease of use in interface design. Saga builds on existing internal tools by offering comprehensive documentation that explains not only how to implement components but also why specific design choices are made. The system is designed to be universal, accessible, flexible, coherent, and distinct, supporting a wide range of configurations and use cases. Grafana Labs encourages community engagement and contributions, maintaining open communication through GitHub and internal channels. The project is committed to continuous improvement, with plans to expand its offerings and introduce new tools to ensure that user experiences remain innovative and seamless.

Grafana Cloud's upgrade from MySQL 5.7 to MySQL 8 was driven by the former's end-of-life and involved transitioning tens of thousands of customer databases across multiple servers and cloud providers. The migration emphasized minimal customer downtime, a small blast radius, and the ability to revert to the previous version if necessary. The team chose to use the mysqldump approach, allowing for selective database migrations with minimal disruption. The upgrade process revealed disk performance bottlenecks, leading to improvements that halved query execution times on busy servers. A phased rollout approach was employed across different environments and customer tiers, monitored through Grafana dashboards to track progress and resource utilization. Challenges such as deprecated MySQL variables, disk I/O bottlenecks, and unusually large databases were addressed, demonstrating the importance of incremental rollouts with the capability for quick rollbacks. The successful upgrade underscored the value of cautious, well-monitored deployment strategies.

KubeCon + CloudNativeCon North America 2023 is set to showcase the latest advancements in Kubernetes Monitoring within Grafana Cloud, highlighting key features such as cost management and resource usage monitoring. The new cost monitoring feature helps manage Kubernetes expenses by offering tools for identifying cost-intensive components, reallocating resources, and providing actionable insights to lower infrastructure costs, all built on the open-source tool OpenCost. Additionally, machine learning capabilities have been integrated to predict CPU and memory usage, allowing users to anticipate potential resource issues, while a comprehensive homepage aids in identifying critical infrastructure issues with ease. The platform also supports an expanded range of integrations for monitoring various services, and continuous optimization efforts aim to enhance efficiency and budget control. Grafana Cloud offers a 14-day free trial for those interested in exploring its observability solutions.

ObservabilityCON 2023, hosted by Grafana Labs, will kick off with an opening keynote on November 14, which will be livestreamed to share insights on the latest trends in open source observability and updates to the LGTM (Loki-Grafana-Tempo-Mimir) stack. The keynote, led by Grafana Labs CEO Raj Dutt and CTO Tom Wilkie, will delve into themes such as observability cost management and the introduction of Adaptive Metrics in Grafana Cloud. In addition to the keynote, sessions will be available on demand, covering topics like unifying application and infrastructure observability, building observability at scale, and the impact of AI and machine learning on the field. Grafana Labs is also conducting an observability survey to gather insights on user practices and challenges, offering participants a chance to win Grafana merchandise.

OpenAI's advancements, particularly with models like GPT-3 and GPT-4, have become integral to various applications, necessitating effective monitoring to manage their complexity and cost. Grafana Cloud offers a seamless integration for this purpose, providing prebuilt dashboards to track OpenAI usage, optimize resources, and ensure application efficiency. This integration aids in cost control by monitoring token usage, enhances resource optimization, and helps in the selection of appropriate AI models. It also supports quality assurance by analyzing prompts and responses, tracks performance metrics such as response times, and identifies usage trends for better capacity planning. The Grafana OpenAI solution includes prebuilt alerts to address specific usage scenarios, ensuring proactive management of potential issues. Users can easily set up this integration with Grafana Cloud by installing a Python library that facilitates the monitoring of OpenAI API usage, thereby enhancing the visibility and performance of AI-driven applications.

Grafana Tempo 2.3 introduces significant enhancements with the vParquet3 backend, which optimizes search speeds by allowing dedicated string attribute columns for more efficient TraceQL queries, while maintaining the previous version as default for smoother upgrades. The release also adds several new features to TraceQL, including structural and negated structural operators, a negated regex operator, and a new intrinsic called statusMessage, alongside comprehensive support for OpenTelemetry attributes, enabling more flexible and complex queries. Future developments aim to expand TraceQL's capabilities to query links, events, and metrics directly from traces, with community feedback invited for these initiatives. Grafana Tempo is accessible through Grafana Cloud's free tier, offering substantial data allowances for traces, logs, and metrics, with community engagement encouraged via various platforms.