March 2023 Summaries

In March, Grafana Labs focused on webinars exploring the core components of the Grafana LGTM Stack, which includes Loki for logs, Grafana for visualization, Tempo for traces, and Mimir for metrics. The sessions covered getting started with Grafana, integrating plugins like Google Cloud Monitoring, and scaling Prometheus metrics using Grafana Enterprise Metrics. The webinars provided practical demonstrations of features such as creating a unified dashboard, managing multi-tenancy with role-based access control, and visualizing external data sources. Upcoming webinars in April and May will prepare attendees for GrafanaCON 2023, focusing on scaling observability, designing dashboards, and advanced querying techniques. These sessions aim to deepen understanding of the Grafana ecosystem, including its new developments like TraceQL for distributed tracing in Grafana Tempo 2.0, and the capabilities of the robust plugin ecosystem.

GrafanaCON 2023 is set to feature the release of Grafana 10, celebrating a decade of the Grafana project, with an engaging agenda including sessions from Grafana experts and community members worldwide. The virtual conference begins on June 12 with a technical workshop, followed by presentations covering new features of Grafana 10, innovative uses in organizations such as NVIDIA and NASA, and home automation with Raspberry Pi. Key sessions will explore Grafana's role in transforming IT and business operations, enhancing security log analysis, and reducing energy consumption in data centers. Additionally, attendees can participate in the Golden Grot Awards, recognizing outstanding Grafana dashboards, with winners receiving a trip to Stockholm.

Grafana has introduced the Golden Grot Awards to recognize exceptional dashboards created by its community, coinciding with the upcoming GrafanaCON 2023 celebrations. The awards will honor three top dashboards in two categories: personal and professional. Community members can submit one dashboard per category by April 21, after which internal judges will shortlist the entries. From May 1 to May 5, the Grafana community will vote for their favorites, with the winning dashboards showcased at GrafanaCON 2023. Grand prize winners will receive a trip to Stockholm to join a special event with Grafana creators. The awards pay homage to Grot, a legendary figure associated with Grafana, and dashboards must be created in Grafana, maintained, and publicly shareable to qualify.

Compliance requirements increasingly affect a wide range of industries, leading companies to retain extensive digital transaction records, which can be costly when using traditional Security Information and Event Management (SIEM) platforms. These platforms not only demand significant staffing and infrastructure expenses but also incur high licensing costs, impacting the total cost of ownership (TCO). To address these challenges, companies are diversifying their log management strategies by using multiple platforms tailored to specific tasks, such as employing Grafana Loki for non-SIEM logs. Grafana Loki offers a cost-effective solution by indexing only key fields rather than full-text logs, reducing the index size and, consequently, the TCO. This allows organizations to allocate resources more efficiently, focusing on cybersecurity investments amid geopolitical tensions, while maintaining compliance through scalable and flexible log management. Despite its advantages, Loki is not ideal for SIEM-style lookups, so companies often integrate it with SIEM solutions via Grafana dashboards, offering a unified interface to manage compliance and security needs effectively.

Grafana Mimir, an open-source time series database (TSDB) introduced by Grafana Labs, has made significant strides in scalability and performance since its launch. Designed to handle over 1 billion active series, Mimir offers features such as easy deployment, multi-tenancy, durable storage, and high availability, while maintaining fast query performance. The platform supports various metrics formats, including Prometheus, Datadog, InfluxDB, and Graphite, without requiring code changes, thanks to its open-sourced write proxies. Mimir's architecture allows for efficient data ingestion and query execution, utilizing sharding techniques to enhance performance. Recent updates include support for out-of-order sample ingestion, broadening its applicability beyond traditional Prometheus limitations. The project, gaining popularity on GitHub, is continuously evolving with plans for expanded capabilities like native histograms and simpler deployment modes. Major companies, such as Adobe and Pipedrive, have adopted Mimir to overcome scalability issues, highlighting its effectiveness in large-scale observability solutions.

Grafana Pyroscope is the result of merging Grafana Phlare and Pyroscope, driven by the collaborative efforts of Dmitry Filimonov, Ryan Perry, Cyril Tovena, and Christian Simon, each bringing their unique experiences and expertise in observability and continuous profiling. In a lively discussion, they share their personal journeys into the tech world, highlighting their excitement about the merger and the potential to enhance observability solutions for users. The conversation also delves into their personal interests, ranging from chess and woodworking to organizing music festivals, and their playful musings on superpowers they'd choose. The merger aims to create a more powerful and seamless observability solution by aligning the projects and integrating continuous profiling into Grafana Cloud, promising to expand the reach and capabilities of profiling technology within the Grafana community.

Grafana has released version 9.4.7, along with security patches for versions 8.5.22, 9.3.11, and 9.2.15, to address a stored XSS vulnerability identified as CVE-2023-1410. This vulnerability was responsibly disclosed through Grafana Labs' Bug Bounty program and involves an issue with the FunctionDescription tooltip in Graphite data sources that could allow attackers to execute XSS payloads. The vulnerability has a medium severity rating of CVSSv3.1 5.7 and can lead to privilege escalation if an attacker gains control over a configured Graphite data source or if a Grafana admin adds a malicious one. Users are advised to upgrade their Grafana installations to mitigate the threat. Grafana Labs encourages reporting of security vulnerabilities via a designated email and provides detailed security announcements and updates through their blog and RSS feed.

Laura Fernández Fernández shares her journey of joining Grafana Labs as a junior frontend developer, reflecting on the transition from her previous role in a small digital marketing company in Spain to becoming part of an international, remote-first organization. Despite initial doubts about her fit and expertise, Laura embraced the opportunity to learn and grow in an open-minded, supportive environment that values transparency and honest communication. At Grafana Labs, she experienced a significant cultural shift, appreciating the flexibility and respect for work-life balance, which allowed her to fulfill personal commitments as a mother. The company’s emphasis on professional development, through internal courses and a yearly learning budget, fueled her passion for continuous learning. Laura describes the welcoming and collaborative culture at Grafana Labs, illustrating her positive experience during an in-person offsite event where she connected with team members, reinforcing her confidence in her decision to join the company.

Grafana Cloud k6 is a newly introduced, fully managed, scalable solution for load and performance testing that aims to enhance the collaboration and efficiency of developers, operations, and QA teams. By integrating performance testing into the development process, Grafana Cloud k6 facilitates a shift-left approach, increasing release confidence and velocity while reducing the time to resolve issues through full-stack visibility. The platform supports scripting tests in JavaScript, integrates with CI/CD tools, and offers the capability to run globally distributed tests with up to 1 million concurrent users. It also provides real-time visualization and analysis, allowing users to correlate k6 test results with server-side metrics, thus improving the identification of root causes without switching platforms. Existing Grafana Cloud customers can access this service, with varying virtual user hours included depending on their subscription, while non-customers have the option to sign up for a free account.

In a journey that spans from aquaponics farming in South Africa to a career at Grafana Labs, Chris Wilkes shares his experiences of managing a fish farm and its unexpected connection to data visualization. Initially driven by a passion for marine ecology, Wilkes embarked on an aquaponics project, combining fish farming and hydroponics to create a sustainable ecosystem. The challenges of this endeavor, including mechanical failures and steep learning curves in IoT, led to the implementation of monitoring systems using Arduino and Raspberry Pi, ultimately integrating Grafana for data visualization. This transformation provided insights into farming operations, revealing previously unnoticed patterns and enhancing efficiency. After selling the farm and transitioning to software engineering in London, Wilkes joined Grafana Labs, finding that the skills and lessons learned in farming have informed his work in software development and monitoring systems. Despite his current role, Wilkes plans to return to farming, motivated by the need for sustainable food production and the potential to leverage data visualization technologies like Grafana in future agricultural ventures.

Kubernetes Monitoring in Grafana Cloud provides a comprehensive solution for observing Kubernetes fleet performance, integrating existing Grafana dashboards and alerts with prebuilt ones to create a unified monitoring platform. This integration allows users to streamline their monitoring operations, reducing the need for multiple interfaces and ensuring a single view for infrastructure health. The process involves migrating existing alerts by changing their namespace and group settings, while dashboards are moved by tagging them appropriately within the Kubernetes Monitoring page. This setup not only consolidates monitoring efforts but also enhances them with Grafana Cloud's additional features and capabilities, offering a personalized and powerful observability ecosystem. Admin access is required to perform these migrations, and users are encouraged to familiarize themselves with Grafana's role-based access controls. The platform also offers a free 14-day trial for new users to explore its full range of features.

Furkan Türkal shares his experience of developing a monitoring solution for his xDSL modem using Prometheus Exporter and Grafana Cloud. Faced with frequent internet outages, he sought to identify whether issues stemmed from his ISP or his modem. By leveraging the SSH capabilities of his ASUS DSL-AC51 modem, Türkal successfully scraped modem metrics, transforming them into a Prometheus-readable format. He utilized the go-dsl library to facilitate this process, enabling him to create a custom Prometheus Exporter to bridge the gap between non-Prometheus-compliant applications and the monitoring platform. Türkal configured Grafana Cloud to visualize these metrics and created alerts using Grafana OnCall to notify him of significant changes, such as drops in the SNR margin. His project not only provides personal insights into his network's performance but also serves as an open-source contribution, inviting further development and use by others interested in similar monitoring setups.

Pyroscope, an open-source continuous profiling project, has been acquired by Grafana Labs, resulting in the merging of Pyroscope with Grafana Phlare under the new name Grafana Pyroscope. This strategic move underscores the belief in continuous profiling as an essential fourth pillar of observability, complementing metrics, logs, and traces. Continuous profiling offers developers enhanced insights into resource usage, aiding in performance optimization and cost management. The collaboration, driven by shared values on developer experience, aims to advance profiling technology and facilitate easier integration of profiles with metrics, logs, and traces in Grafana Cloud. This merger is expected to enhance the capabilities of engineering teams in swiftly identifying and resolving issues, thus improving software delivery.

Integrating legacy systems with modern monitoring tools can be challenging due to the presence of personally identifiable information (PII) and IP addresses in log files. Grafana Cloud simplifies this process by enabling the monitoring of legacy systems like QMailToaster while protecting user privacy and enriching logs with geolocation data. Utilizing tools such as Promtail and Grafana Agent allows for the ingestion and parsing of logs, obfuscating sensitive information, and performing GeoIP lookups to add geographic labels, resulting in enhanced data analysis. By implementing Grafana Cloud Synthetic Monitoring, users can also generate valuable SLO/SLI indicators, providing insights into the health and performance of email services, identifying potential issues, and ensuring compliance with privacy regulations. This approach not only saves time and maintains data privacy but also offers businesses a comprehensive view of their system's reliability and user distribution, contributing to improved service standards.

Grafana Agent v0.32 introduces several new integrations and enhancements, making it easier to monitor infrastructure with expanded support for Oracle Database, MS SQL Server, AWS Cloudwatch, Azure Monitor, and GCP Cloud Monitoring. The release marks the end of the beta phase for Flow, a configuration mode that enhances the agent's functionality and ensures full compatibility with OpenTelemetry, thereby enhancing application observability through metrics, logs, and traces. New Flow components have been added for existing exporters like prometheus.exporter.apache and prometheus.exporter.redis, and users can leverage these tools to monitor system usage, such as Firefox resource consumption. The update also includes changes to component naming conventions, the merging of some Flow components to streamline hierarchy, and the temporary removal of Flow UI support for 32-bit ARMv6 builds. Users are encouraged to explore the comprehensive documentation and tutorials provided by Grafana Labs to effectively utilize these new features and participate in the community for feedback and support.

Grafana's comprehensive monitoring capabilities for AWS Fargate and other serverless environments are highlighted through its ability to seamlessly correlate metrics, logs, and traces, making it easier to troubleshoot and maintain these dynamic infrastructures. Grafana's "big tent" philosophy supports diverse data sources, enabling users to visualize and manage cloud costs, resource inventories, and application performance metrics. The Grafana Agent simplifies the process by collecting telemetry data and sending it to Grafana Cloud, offering users a single platform for monitoring and alerting. This approach provides a more granular view than native AWS solutions, allowing users to monitor applications, containers, and Fargate infrastructure effectively. Additionally, Grafana's integration with various sources like Amazon CloudWatch and SQL databases enhances its utility, offering contextual layouts and visualizations that help identify and resolve issues quickly by providing a correlated view of the environment.

Grafana Cloud is introducing a revamped navigation experience, including enhancements to the command palette, designed to streamline workflows and improve user efficiency. The command palette, accessible via keyboard shortcuts, allows users to perform actions and navigate through the platform with ease, offering features like direct dashboard creation and access to recently viewed dashboards. These updates, which will become default in Grafana OSS and Grafana Enterprise with the Grafana 9.5 release, aim to make Grafana more intuitive and accessible, particularly for power users. Additional improvements include an optimized Grafana Loki querying experience and updates to Grafana Alerting to simplify alert management. Users are encouraged to explore these new features and can take advantage of a free 14-day trial of Grafana Cloud Pro to experience the platform's full capabilities.

Observability is gaining traction among SREs and DevOps practitioners as they aim to centralize tools and data sources to manage workloads efficiently and respond to incidents promptly. The Grafana Labs Observability Survey 2023, which gathered insights from over 250 practitioners, highlighted the shift from traditional monitoring to modern observability practices due to the complexity of today's applications and infrastructure. The survey revealed that most companies use multiple observability tools, with some employing over 75, and highlighted a spectrum of adoption across industries, with financial sectors leading in centralized observability implementation. Respondents noted significant time and cost savings from consolidated observability, with many companies exploring service level indicators and objectives to enhance accountability and market maturity. The survey also found that while saving money is a universal aim, organizations pursue various strategies to achieve return on investment, such as improving mean time to resolution and reducing complexity. The results, along with in-depth interviews with Grafana Labs customers, reflect a positive outlook on the observability market's growth and maturation, emphasizing a unified approach to solution integration to reduce mean time to recovery and improve developer productivity.

Grafana 9.4 introduces several enhancements aimed at improving the Loki querying experience, including query validation, improved autocomplete, and integrated query history. The update features real-time query expression validation, which identifies errors with a "red squiggly line," and supports multi-line queries. Autocomplete suggestions have been refined to provide better quality, including parser type recommendations. A new query history feature in the code editor helps users leverage past queries, while an estimated query size feature informs users about the potential data processing impact. The label browser has been updated for better accessibility, and a logs sample feature allows users to see contributing log lines for metric queries, aiding debugging. Additionally, new log details functionalities enable more precise filtering without complex operations, and a line limit display helps users understand the maximum log lines visible in results. These improvements aim to enhance efficiency and user experience in data exploration and query composition.

Since the release of Grafana 9.0, Grafana Labs has implemented numerous improvements to Grafana Alerting, driven by user feedback and aimed at enhancing user experience in creating and managing alerts. These updates include a new landing page, a redesigned expressions pipeline, improved alert rule search capabilities, and the ability to copy existing alert rules. Users can now export alert rules for provisioning, better understand evaluation intervals, and enjoy more intuitive action buttons. Additionally, a dashboard and panel picker have been added to streamline rule creation, and incidents can be created directly from firing alerts in Grafana Cloud. Troubleshooting notification errors is now easier with enhanced feedback mechanisms, and the documentation for templating notifications has been rewritten to include tutorials and examples. Looking ahead to Grafana 9.5, further enhancements are planned to continue improving the user experience.

Kubernetes Monitoring for Grafana Cloud offers a solution to optimize resource utilization by addressing the challenges of overprovisioning and underprovisioning in Kubernetes environments, which can lead to costly cloud expenditures and performance issues. The tool helps align resource allocation with actual usage, mitigating waste and ensuring enough resources are available to prevent application instability. It identifies unused and stranded resources, allowing organizations to improve resource management, enhance development productivity, and reallocate savings to expand teams. The platform provides insights through dashboards and metrics, allowing for better decision-making regarding resource utilization across multiple clusters and cloud providers. The solution includes features like the Cluster overview and Deployment and Pod landing pages to track resource usage, helping to maintain performance while optimizing costs. By offering a comprehensive view of resource management, Kubernetes Monitoring in Grafana Cloud aids in raising awareness and accountability within organizations, ultimately contributing to improved performance, stability, and cost efficiency.

Grafana Labs offers a series of live and on-demand webinars designed to educate users on leveraging the Grafana LGTM Stack, which comprises Loki for logs, Grafana for visualization, Tempo for traces, and Mimir for metrics. These sessions cover a variety of topics such as enhancing dashboard design, unifying data with plugins, and reducing mean time to identify and resolve issues (MTTI/MTTR) using Grafana's capabilities. Featured webinars include demonstrations on integrating Datadog, Splunk, and MongoDB, visualizing Microsoft Azure and Google Cloud, and starting with Grafana dashboard design. Upcoming sessions will focus on scaling observability, introducing metrics with tools like Prometheus and Graphite, and further exploring plugin utilization, all aimed at providing participants with the tools to optimize their use of Grafana for centralized observability and data analysis.