December 2023 Summaries

InfluxDB and Grafana are key tools for managing and visualizing time series data, often used together to create insightful dashboards. Users must choose from three query languages—InfluxQL, SQL, and Flux—each offering distinct capabilities and suited to different needs based on the InfluxDB version and user expertise. InfluxQL is SQL-like and familiar to many, SQL has been enhanced with the FlightSQL plugin for broader relational database compatibility, and Flux, a functional scripting language, offers extensive functions for data manipulation. Grafana dashboards can effectively utilize these languages to visualize data, such as U.S. electricity metrics, through various chart types and Grafana transformations. Each language has unique syntax requirements for tasks like aligning time ranges and using template variables, while also providing different aggregation capabilities. Ultimately, the choice of language depends on the specific data analysis requirements and the version of InfluxDB being used.

In this installment of "The Concise Guide to Grafana Loki," the author shares best practices to enhance query performance in Loki, a log aggregation system. The guide emphasizes the importance of running Loki in scalable deployment modes like SSD or microservices for improved query performance and recommends upgrading to the latest version to leverage ongoing performance enhancements. Key suggestions include using the TSDB index type for better sharding capabilities, employing Snappy for chunk encoding to boost speed, and utilizing major cloud provider object storage for optimal performance. It also highlights the significance of configuring results and chunk caches to minimize redundant processing and improve I/O operations. The article provides insights into Loki's query execution process, including splitting queries into time segments and shards, and offers troubleshooting advice for performance issues, such as adjusting CPU and memory settings, reducing excessive use of labels, and avoiding complex regular expressions. The piece concludes by identifying common causes of poor query performance, such as improper parallelism settings and inadequate resources, while encouraging the use of Grafana Cloud for monitoring and optimization.

Integrating Grafana Alerting with Telegram allows users to receive immediate notifications directly through the Telegram app, providing a convenient alternative for personal alerts compared to team-based notifications typically sent via platforms like Slack. To set up this integration, users must first create a Telegram bot and obtain its API token and the chat ID where notifications will be sent. These credentials are then used to configure a contact point within Grafana Alerting, which can be linked to a notification policy to ensure alerts are routed to Telegram. The process involves setting up the bot through Telegram's BotFather, obtaining the necessary chat ID via an HTTP request, and configuring Grafana's notification settings to include the new Telegram contact point. This integration is compatible with Grafana OSS, Enterprise, and Cloud, offering a flexible alert management solution for different user needs.

In 2023, Grafana dashboards showcased a diverse array of innovative applications, celebrating their tenth anniversary with the inaugural Golden Grot Awards to honor remarkable user contributions. Notable projects included a UCLA rocket engineering team using Grafana for visualization during competitions, which led to a world record, and a student from the Erasmus School of Economics creating a dashboard to monitor European energy infrastructure. In the professional realm, Raymond Sowden developed a dashboard to enhance observability for Oracle's JD Edwards ERP systems, improving efficiency and insight across various organizational levels. Personal projects saw Grafana used for home monitoring, including air quality, furnace usage, and water levels, while environmental initiatives focused on reducing carbon emissions and balancing national power grids. Grafana's role extended to unexpected areas, such as aiding sustainable farming systems and appearing in pop culture through video games like Call of Duty: Modern Warfare 2, highlighting its versatility and widespread influence.

In 2023, Grafana Labs marked significant milestones, celebrating the 10th anniversary of Grafana and seeing Grafana Cloud become its main revenue driver, a testament to its growing adoption for observability strategies. Grafana Cloud has evolved into a preferred solution due to its ease of use, offering features like continuous profiling and Adaptive Metrics to enhance cost efficiency and system insights. The company emphasized interoperability, allowing users to integrate various data sources without being confined to a single provider. Notable advancements included the introduction of Grafana Cloud Profiles and Grafana Cloud k6 for scalable load testing, enhancing the observability landscape. The release of Grafana SLO and advancements in Grafana IRM highlighted Grafana's commitment to improving service level objectives and incident response management. Furthermore, the launch of Frontend and Application Observability aimed to streamline application monitoring and management, while expansions in the Free tier and Enterprise plugin access underscored Grafana's dedication to accessibility and comprehensive monitoring solutions.

In 2023, Grafana Labs celebrated a decade of open source success, marked by milestones such as the release of Grafana 10 and the launch of new projects like Grafana Pyroscope and Beyla. The Grafana LGTM Stack saw significant updates, with enhancements to Grafana Loki, Grafana Tempo, and Grafana Mimir aimed at improving scalability, performance, and user experience. The year also witnessed the introduction of the Grafana Champions program to recognize community contributions. Grafana Labs maintained its active involvement in the OpenTelemetry and Prometheus communities, bolstering interoperability and collaborative innovation. Overall, 2023 was a transformative year, showcasing Grafana Labs' commitment to evolving its open source offerings and fostering a global community.

Grafana Loki, a logging database, uses labels as both the keyspace for sharding incoming data and the index for log retrieval at query time, making their correct usage crucial for successful application. Labels are key-value pairs assigned to logs, and while they seem simple, they can cause issues if misused, particularly with high cardinality, which Loki is not designed to handle due to its preference for long-lived streams and low cardinality. Recent updates include automatic stream sharding for high-volume streams and structured metadata for storing high cardinality data without indexing it, while Bloom filters are being developed to optimize "needle in a haystack" queries. The guide emphasizes using few labels with long-lived and intuitive values, avoiding ephemeral values like trace IDs, to maintain efficient log querying and indexing, and highlights improvements made in Loki's ability to handle large data volumes and reduce operational challenges.

In 2023, Grafana Labs marked its 10th anniversary with significant milestones, including the release of Grafana 10 and the debut of the Golden Grot Awards, celebrating community contributions. The company expanded its open-source offerings with Grafana Pyrsocope and Grafana Beyla, and enhanced its Grafana Cloud solution to include frontend and application observability, cost management tools, and more. Key developments included the launch of Grafana Tempo 2.0 with TraceQL, the acquisition of Asserts.ai for improved observability insights, and the introduction of Grafana Pyroscope for advanced profiling. Grafana Labs also celebrated the fifth anniversary of Grafana Loki and premiered a documentary series detailing its journey. The company's achievements were recognized with a rise on the Forbes Cloud 100 list and other industry accolades, underscoring its influence and innovation in the cloud and observability sectors.

OpenTelemetry is a framework that has gained traction in the observability community, and Grafana Labs is fully committed to integrating it within their open-source projects and products. The guide outlines best practices for implementing OpenTelemetry, such as starting with auto-instrumentation for ease and gradually adding manual instrumentation for more detailed insights. It emphasizes the importance of initializing OpenTelemetry correctly, generating meaningful attributes, and adhering to semantic conventions to ensure consistent data. The document also discusses strategic data correlation, batching, and sampling to optimize performance and cost-effectiveness. Grafana, a leading contributor to both Prometheus and OpenTelemetry, suggests leveraging span metrics for easy analysis and highlights the benefits of using the OpenTelemetry Collector for data management. The guide also encourages the use of Grafana Cloud for visualizing and correlating OpenTelemetry data, offering prebuilt dashboards and the potential for custom analytics through Grafana Machine Learning.

Grafana k6 version 0.48.0 introduces significant updates, including a new subcommand for generating test scripts and a memory-efficient experimental file system module, enhancing file handling in test scripts by reducing memory usage. The release also brings updates to k6 browser with CPU and network throttling features, simulating slower devices and network connections. Grafana Cloud k6 offers new user-defined project limits to prevent unexpected charges and improvements to distributed tracing, while Private Load Zones v2 is in public preview to simplify internal service testing within Kubernetes clusters. The k6-operator v0.0.12 release enhances the PrivateLoadZone workflow and removes DNS restrictions, alongside community-driven updates to various k6 extensions, including structured logging, protocol support, and new testing capabilities. The Grafana k6 Champions Program continues to recognize community contributions and expertise, highlighting individuals like Paul Maxwell-Walters and their efforts in organizing meetups and advocating for k6. Community members have contributed to enhancing the k6 ecosystem through events, articles, and tools, showcasing the project's versatility and collaborative spirit.

Grafana Incident Response & Management (IRM) has introduced a new capability aimed at enhancing communication and efficiency in handling incidents by integrating Grafana OnCall and Grafana Incident more closely. This feature allows users to directly notify individuals or teams about incidents through their preferred communication channels, such as Slack or email, and includes the ability to specify the severity of an incident. The system ensures timely communication by triggering escalation chains if initial notifications are not acknowledged. This functionality streamlines the process of involving necessary parties in incident management, thereby reducing manual tasks and improving response times. The update is a part of Grafana's ongoing effort to provide a more seamless experience across its suite of tools, including Grafana Alerting, Grafana Incident, Grafana OnCall, and Grafana SLO. Users can access this feature through Grafana Cloud, which offers a forever-free account option to get started with their services.

"The Story of Grafana" documentary series explores the profound impact of Grafana's open-source community, highlighting its collaborative and inclusive nature that has driven the project's success. Founded by Torkel Ödegaard, Grafana has evolved from a simple tool for visualizing time series data into a global phenomenon with over 20 million users. The community's spirit is embodied by contributors who range from fixing minor errors to developing significant features like AWS CloudWatch support, illustrating the project's "big tent" philosophy that encourages interoperability. The documentary, featuring interviews with key figures such as Grafana Labs co-founders and early contributors, underscores the project's growth and the enduring commitment of its members. As episodes release, they delve into themes of open-source business and innovation, inviting viewers to engage with the vibrant Grafana community.

Grafana Loki, introduced five years ago, has grown into a popular open-source log aggregation tool, designed to efficiently handle the chaotic nature of logs without predefined schemas by storing everything as strings and adding structure at query time. It leverages Prometheus labels and the distributed system design of Cortex, making it highly scalable and Kubernetes-friendly. The tool's simplicity and adaptability, alongside innovations like the Promtail agent for easy log categorization, have contributed to its widespread adoption. Key features include a small index size, object storage usage, and a query engine that supports various formats through LogQL. Though initially focused on row-oriented storage, Loki is exploring columnar storage for better performance with structured data. It continues to evolve with better log ingestion, OpenTelemetry support, and enhancements in log indexing using Bloom filters. The tool’s success is attributed to its flexibility, scalability, and ability to cater to diverse logging needs, from homelabs to large multi-tenant systems.

Grafana 10.2 introduces significant enhancements to Grafana Alerting, including the new Insights section on the Grafana Alerting home page, which is available to all Grafana Cloud users. Insights provides valuable data on alert rules and notifications, allowing users to monitor alert performance and identify trends and potential issues. It offers predefined panels categorized into subsections for different data sources, such as Grafana-managed and Mimir-managed alert rules, as well as Grafana and Mimir Alertmanager. Users can customize time ranges and enable real-time auto-refresh to better analyze their alerting data. This update is part of Grafana's ongoing efforts to improve alert management visibility, which includes features like an alert state history system leveraging Grafana Loki. Users are encouraged to provide feedback on the utility of different panels to aid further improvements. Grafana Alerting is accessible via Grafana OSS, Enterprise, or Cloud, with a free tier available for new users.

In the realm of observability, Grafana Tempo's updated features, including "Aggregate by," enable more effective trace analysis by allowing users to compute RED (Rate, Error, Duration) metrics at query time rather than storing massive amounts of data in a time series database. This approach helps avoid the pitfalls of cardinality explosion, which can be costly and inefficient, by facilitating on-the-fly metrics computation tailored to specific investigative needs. The "Aggregate by" feature enables users to break down traces by various attributes, providing a granular view of system performance and aiding in pinpointing issues like elevated error rates or latency confined to particular service regions or endpoints. These advancements are part of a broader strategy to integrate ad hoc metrics computation directly into TraceQL, offering greater flexibility and insight without the overhead of persistent metrics storage. Early adopters like Sarwa have praised the feature for its ability to swiftly identify problematic endpoints, demonstrating its practical utility in real-world scenarios. Grafana Labs is committed to expanding these capabilities, aiming for more comprehensive trace metrics analysis in future updates.

Milan Plžík, a Senior Software Engineer at Grafana Labs, argues for the case of Kubernetes resource limits by examining the trade-off between predictability and efficiency in workload performance. While some advocate against using resource limits to maximize compute efficiency, Plžík suggests that setting limits can prevent performance unpredictability and help manage resource allocation more effectively, especially during high-traffic periods like Black Friday and Cyber Monday. He discusses strategies such as configuring workload limits to be slightly above requests, known as fixed-fraction headroom, or setting requests equal to limits for predictable performance. These approaches ensure a balance between resource utilization and stability, although they may lead to some resource wastage. Plžík emphasizes that while configuring limits is challenging and can lead to inefficiencies if done incorrectly, it provides valuable insights into workload behavior under stress and can help prevent catastrophic failures. He highlights that the decision to set limits should consider various factors, including workload type and resource availability, to achieve an optimal balance between resource efficiency and performance predictability.

"The Story of Grafana" is a four-part documentary that celebrates the 10-year journey of Grafana, a data visualization software originally created by Torkel Ödegaard in 2013, which has grown into a prominent open-source project with over 20 million users worldwide. Directed and produced by Collins Pace, the documentary features more than 20 interviews with key figures, including Ödegaard, co-founders Raj Dutt and Anthony Woods, and prominent contributors from the global open-source community. The first episode, "Democratize Metrics," delves into Grafana's early days and its initial growth, highlighted by Ödegaard's 14-day commit streak and the first GrafanaCON in 2015. The documentary aims to illuminate the community, the business of open source, and the innovations driving Grafana's future, while also sharing anecdotes like the origin of Grafana's name. Despite its growth into a tool used for diverse applications, such as NASA rocket launches and pet monitoring, Ödegaard remains deeply involved in the project, consistently working on improvements and engaging with the community.

Microsoft Windows' integration with Grafana Cloud has been enhanced to provide a more comprehensive and user-friendly experience for monitoring environments that include both Windows and Linux systems. The updates include a synced dashboard style with Linux, allowing for a consistent look and feel, which simplifies management across mixed operating systems. Key improvements feature a three-tier view for monitoring, enhanced navigation, and new alerts to offer better control over resources. The fleet view dashboard now allows users to monitor multiple virtual machines (VMs) at scale, providing a unified perspective to spot issues efficiently. Additionally, the integration introduces annotation integration for event correlation and utilizes modern tools like Jsonnet for modularizing dashboards, ensuring a cohesive user experience. These enhancements aim to streamline VM management in Grafana Cloud, making it easier for engineers to monitor and address potential issues across their infrastructure.

Grafana Cloud offers a fully managed observability stack, enhancing log management by allowing companies to focus on innovation rather than maintaining self-hosted systems. Observability experts from Blinkit, Paradigm, Houzz, and Exabeam share their experiences of migrating from open-source solutions to Grafana Cloud, highlighting benefits such as streamlined log management, reduced operational overhead, and improved data correlation. Blinkit, an instant grocery delivery service, integrated Grafana Cloud to efficiently manage 60TB of log data monthly, while Paradigm leveraged it to simplify their logging processes with a small team. Houzz benefitted from the ability to cross-reference logs with metrics for quicker issue resolution, and Exabeam centralized its observability efforts with Grafana’s cloud-native tools to support their extensive cybersecurity services. Each case reflects how Grafana Cloud helps organizations better manage logs and metrics, providing robust support and enabling more proactive system monitoring and troubleshooting.

Grafana Cloud has introduced several new features aimed at enhancing end-to-end observability, as highlighted during ObservabilityCON 2023. These updates, including Grafana Cloud Application Observability, Grafana SLO, and Adaptive Metrics, are designed to improve root cause analysis, manage service level objectives, and optimize costs without sacrificing observability capabilities. Expert-led sessions provide insights into these features and offer best practices for their implementation, while user stories showcase successful applications of Grafana Cloud in diverse environments. Notably, Grafana Cloud's capabilities enable organizations to effectively correlate application and infrastructure observability and manage metrics and logging costs. Additionally, customer case studies reveal how companies like Actian and ComplyAdvantage have leveraged Grafana Cloud to transform their observability strategies through customizable tools and increased OpenTelemetry usage. Grafana Labs encourages participation in its observability survey to gather insights into industry practices and challenges.