December 2022 Summaries

In 2022, Grafana dashboards were utilized in various innovative ways, showcasing their versatility in monitoring diverse projects. Notable applications included an Ambient Assisted Living system designed by Amon Reich for monitoring his elderly parent's home, which provided a cost-effective alternative to hiring aides. Additionally, Chris Field and Mikhail Volkov leveraged Grafana with machine learning for real-time microscopy image analysis, significantly reducing labor costs. Matt Stewart used Grafana to monitor his Tesla Model 3, while Ivana Huckova tracked her sourdough starter's growth using Grafana Mimir. Grafana was also employed by Willie Engelbrecht to monitor a hard water softener and by Paul Leroy and his daughter to ensure their pet python's environment was optimal. Other uses included visualizing fitness metrics, monitoring high-performance computing systems at NERSC, enhancing agricultural efficiency, supporting maritime operations, and improving uptime for Dish Network's satellite communications. Grafana also facilitated data visibility at large events like Cisco Live and helped Optum monitor website performance, demonstrating its broad applicability across sectors.

In 2022, Grafana Cloud significantly enhanced its offerings by integrating and upgrading its open-source LGTM stack, which includes Grafana Loki for logs, Grafana for visualization, Grafana Tempo for traces, and Grafana Mimir for metrics. This was aimed at providing an easy-to-use and integrated platform for users without the time or expertise to manage a DIY approach. Key developments included a comprehensive Kubernetes Monitoring solution, a unified Incident Response Management (IRM) stack featuring Grafana OnCall and Grafana Incident, and expanded Grafana Alerting capabilities with machine learning enhancements. Grafana Cloud also increased its global availability and expanded its catalog of integrations, including strategic partnerships and new support for various logs, to facilitate widespread adoption of observability best practices. These improvements were designed to help users efficiently manage and monitor their infrastructure, respond to incidents, and implement observability across their organizations.

In 2022, Grafana Labs made significant strides in the open-source space by acquiring Pyroscope and integrating it with Grafana Phlare, and completing their observability stack with Grafana Mimir for metrics. They continued to enhance their flagship projects, including Grafana Loki, Grafana Tempo, and the core Grafana application, with various upgrades to improve scalability, accessibility, and functionality. New projects like Grafana OnCall, Grafana Phlare, and Grafana Faro were introduced to address on-call management, continuous profiling, and frontend application observability, respectively. Grafana Labs remains committed to supporting the open-source community, contributing to projects such as OpenTelemetry and Prometheus, and advocating for interoperability within the observability ecosystem. The company's efforts are driven by community feedback and a dedication to advancing open-source software, as evidenced by their ongoing updates and innovations across their product suite.

In 2022, Grafana Labs marked a significant year with numerous achievements and developments, including surpassing one million active Grafana instances, launching four new open-source projects—Grafana Mimir, Phlare, Faro, and OnCall—and releasing Grafana 9.0, which focused on enhancing user experience for data visualization and observability. The company also secured $240 million in Series D funding, welcomed strategic partnerships with Isovalent and Google Cloud, and hosted in-person community events worldwide, such as GrafanaLive and ObservabilityCON. Additionally, Grafana Labs expanded its workforce to over 850 employees, was recognized on the Forbes Cloud 100 list, and launched "Grafana's Big Tent" podcast, all while maintaining a strong commitment to open-source development and community engagement.

The Grafana k6 Champions Program has been launched to acknowledge and support performance testing experts who contribute to the k6 community. This initiative aims to recognize individuals passionate about mentoring and sharing their knowledge of performance testing and k6, whether through events, tutorials, or community interactions. The program started with five inaugural champions, each distinguished by their contributions to the field and community engagement. Participants in the program are expected to contribute four activities annually, which could include blog posts, workshops, or speaking engagements, and they receive various perks such as free access to k6 cloud services and exclusive merchandise. The program encourages global applications, particularly from diverse and underrepresented communities, providing a platform for advocates to strengthen their personal brands and collaborate with like-minded individuals.

Outlier Detection is now a feature within the Grafana Machine Learning toolkit available on Grafana Cloud for Pro and Advanced users, designed to monitor groups of similar entities, such as Kubernetes pods, and alert users when anomalies occur. This tool is particularly useful for managing the complexity of horizontally scaled applications by comparing the behavior of similar pods to identify deviations, helping to address issues like higher error rates early. By using algorithms like DBSCAN and MAD, users can detect outliers by comparing current data with historical trends or identifying clusters, respectively. The feature integrates with Grafana Alerting to notify users of detected anomalies, and it allows for the adjustment of sensitivity settings to tailor outlier identification to specific needs. Grafana Cloud offers a free trial and ongoing support through its community channels, inviting users to explore data insights and make use of its comprehensive monitoring capabilities.

Torqata, a SaaS startup aiming to revolutionize the tire sales industry by enhancing data visibility, utilizes Grafana Alerting to streamline its data health scorecard system. The company employs a comprehensive data processing architecture hosted on Google Cloud, encompassing data ingestion, ETL microservices, and data storage solutions like data lakes or warehouses. By leveraging Grafana, Torqata offers a dynamic scorecard dashboard that provides aggregate statuses and deeper data analyses, allowing retailers to gain insights into pricing and inventory needs. The system's scalability is enhanced through automated onboarding and multi-dimensional alerting rules, facilitating the integration of new data feeds without manual intervention. This approach not only reduces onboarding time significantly but also ensures data quality and proactive issue resolution, thereby minimizing downtime and enhancing customer experience. The alerting system, which Torqata has successfully used in production for over a year, allows for efficient management of alerts and offers a holistic view of data quality by linking multiple data sources through Grafana plugins.

Grafana Agent v0.30 introduces significant updates, including the beta graduation of Agent Flow and the addition of Loki components for logging pipelines, enhancing the tool's capabilities in handling telemetry data. The release supports seamless integration between Prometheus and OpenTelemetry ecosystems, aligning with Grafana's Big Tent philosophy, and aims for feature parity with the static Agent mode by incorporating native histograms and improvements to the Grafana Agent Operator. The new logging pipeline features allow for the discovery, transformation, and shipping of log entries using components like discovery.file, loki.source.file, loki.relabel, loki.process, and loki.write. The update also includes plans for future enhancements, such as converting logs between Loki and OpenTelemetry, with a continued focus on performance, stability, and community engagement.

Grafana Labs has introduced a new comprehensive Helm chart for deploying Grafana Loki, consolidating previous charts to streamline installation and management for both open-source and Enterprise deployments. This new chart, available from Helm version 3 or higher, supports both scalable and single-binary deployment modes, simplifying the process for users by becoming the primary chart for all Helm-deployed Loki clusters. It includes advanced features like Loki Canary and Self-Monitoring, which automatically configure log and metric scraping, ensuring robust monitoring and production-ready observability with pre-packaged Grafana dashboards. The chart also facilitates smoother integration with Grafana Enterprise Logs by allowing tenant provisioning during installation, addressing previous issues of update delays and configuration difficulties. Originating from community contributions, the new chart aims to provide a clear starting point for users, encouraging feedback and continued community involvement while shifting Grafana Labs' focus to this unified chart to enhance user experience and simplify the deployment process.

Machine Learning Reply, part of the global Reply network, is a company that focuses on providing innovative business solutions centered around AI, ML, and other advanced technologies. The company values a combination of technical and interpersonal skills in potential hires, seeking individuals who are not only technically proficient but also capable of effective communication and agile thinking. The team's dynamic emphasizes open communication, collaboration, and continuous learning, with an open-door policy that fosters a supportive environment. Observability is a key component of many solutions, with knowledge of tools like the ELK-Grafana-Prometheus stack being highly valued. The company supports its employees in acquiring new skills through training and certifications, promoting a culture where readiness to learn is as important as technical expertise. The leadership style at Machine Learning Reply is characterized by encouraging initiative and personal growth, allowing team members to explore new challenges and expand their skill set.

Nick Moore's guide delves into the process of cyber threat hunting using a combination of tools, including Promtail, Grafana Loki, Sigma, and Grafana Cloud, to enhance log observability and detect potential intrusions. By utilizing Sigma's structured format for identifying suspicious activities within log files, the guide demonstrates how to translate Sigma rules into Grafana Loki queries with the help of the pySigma project. This enables security teams to examine logs for indicators of compromise (IOCs) effectively. The guide provides a step-by-step approach to setting up the necessary environment, starting with generating Windows log data using Sysmon, collecting it with Promtail, and storing it in Loki for analysis. Furthermore, it explains the installation and configuration of these tools, showcasing how to convert Sigma rules into Loki queries that can be used for searching logs within Grafana Cloud. The guide also hints at future capabilities, such as integrating Sigma rules with Grafana Alerting for real-time notifications and expanding the use of these tools across various systems to monitor security incidents.

KCB Bank Uganda significantly enhanced its transaction service monitoring by implementing Grafana, transitioning from outdated Excel-based processes to a centralized and efficient monitoring tool. This shift allowed the bank to improve service uptime through real-time alerts and comprehensive data visualization, capturing data from various sources such as Microsoft SQL Server and JSON. As a result, the bank achieved a 96% improvement in transaction service monitoring, leading to consistent revenue collection and increased operational efficiency. Inspired by weather-related alerts, the bank designed dashboards for early warning metrics, which contributed to a more resilient and scalable banking operation. This successful implementation opened opportunities for expansion into areas like fraud detection and real-time transaction trend analysis, while also allowing IT staff more flexibility. The ongoing improvements aim to further enhance customer experience and operational convenience, as highlighted by Systems and Software Engineer Fauzi Lutaaya in a GrafanaCONline 2022 session.

Grafana has announced a revised release schedule to address the challenges posed by its rapid growth and frequent updates, which have increased by 70% over the last four years, resulting in 65 releases in a single year. Starting January 10, 2023, Grafana will adopt a monthly release cycle for self-managed instances, with minor versions featuring new features and security updates on even-numbered months and patch releases focusing on bug fixes and security updates on odd-numbered months. This change aims to provide a more predictable and reliable update experience for users, while still allowing for rapid responses to high-severity security incidents. Grafana Cloud users will remain unaffected by this shift, and the company will maintain its commitment to providing seamless service continuity. Additionally, a release freeze is in place from December 15, 2022, to January 4, 2023, to accommodate the holiday season, with exceptions for critical operational or security incidents.

Sandwich view is a mode of viewing flame graphs that was popularized in the Speedscope project by Jamie Wong, and it effectively separates a flame graph into two sections: callers and callees, providing a clearer understanding of resource utilization within functions. This mode offers an advantage over the typical flame graph by highlighting the time spent in a function itself versus the time spent in the functions it calls, thus facilitating the identification of optimization opportunities. For instance, while a standard flame graph might highlight a single function like rideshare/car.OrderCar as a point of optimization due to its large width, the sandwich view can reveal frequently called functions like Time.Since() even if they have a shorter width, across various code paths. To utilize the sandwich view, users can access it through the Pyroscope UI or flamegraph.com by selecting the appropriate icon, and Grafana Labs, which acquired Pyroscope in 2023, plans to introduce more view modes in the future.

Grafana 9.3 introduces a new command in the Grafana CLI designed to address user identity conflicts caused by case-sensitive logins and emails, which can result in multiple accounts for the same user and related complications. This tool allows administrators to manage these conflicts by listing conflicting user accounts, generating a conflict file for review, validating changes, and ingesting the resolved file to update the database, with the last step being irreversible. The tool helps streamline user management by resolving discrepancies and preventing the creation of "zombie" accounts or split user permissions. While this feature supports ongoing efforts to enhance Grafana's consistency, it will be phased out by October 2024, encouraging admins to utilize it for versions up to Grafana 11.4. For those running Grafana Cloud or using MySQL, these conflicts are not encountered, and further configuration settings can ensure future conflicts are avoided.

Formula 1 teams leverage real-time analytics to gain critical insights and make data-driven decisions during races, with a significant role played by Azure Data Explorer (ADX) and Grafana Cloud. Anshul Sharma, a Senior Product Manager at Microsoft, highlighted the importance of telemetry data in F1, where cars equipped with 300 sensors generate 1.1 million data points per second, translating to 160 TB per race. This data is crucial for optimizing car performance, detecting issues, and making in-race adjustments. In his presentation at GrafanaCONline, Sharma demonstrated how telemetry data from a simulated F1 race can be processed and visualized using ADX for analytics and Grafana for advanced dashboards, enabling teams to monitor metrics like tire wear and engine temperature in near real-time. The integration of ADX with Grafana allows for seamless data visualization through a user-friendly interface, supporting interactive queries at petabyte scale. This analytics stack, which can be applied beyond F1 to industries like retail and finance, enables high availability and minimal latency by deploying Grafana Cloud in the same Azure region as ADX.

Grafana 9.3 introduces enhanced OAuth token management to improve security and user experience by ensuring users are continuously authorized through their OAuth identity provider. This update, available in Grafana OSS, Cloud, and Enterprise, verifies that a user's access token is valid on each request, utilizing a refresh token to obtain a new access token when necessary, thereby avoiding interruptions in user sessions. This feature requires configuration of the identity provider to use refresh tokens, and it can be enabled with the accessTokenExpirationCheck feature toggle for Grafana setups. The update supports various OAuth providers like Azure AD, Okta, and Google, and aims to maintain secure and seamless access to Grafana dashboards. Additionally, Grafana 9.3 includes new navigation and localization features, with further details available in the release changelog and documentation. Users are encouraged to explore these changes through community forums and support resources.

Grafana 9.3 introduces new navigation updates aimed at enhancing its usability as it evolves from a visualization platform to a comprehensive observability solution. The updates focus on making the platform more intuitive for new users and improving efficiency during incident response by implementing consistent page layouts and a revamped navigation menu that groups related tools for easy access. These changes include features such as breadcrumbs, a sidebar for quick page transitions, and a new header with a dashboard search function. Users can enable the new navigation in beta by using the topnav feature toggle for on-premise installations or by contacting support for Grafana Cloud. The release encourages feedback through GitHub discussions and offers resources like changelogs, community forums, and webinars to help users integrate the new features into their workflows.

Managing Grafana as code offers numerous advantages, such as improved workflows, code review, and reuse, by enabling a declarative approach to configuring resources. This guide explores several tools for managing Grafana as code, including the Grafana Terraform provider, Grafana Ansible collection, Grizzly, Grafana Crossplane provider, and the Kubernetes Grafana Operator. Each tool has its benefits and limitations, catering to different user needs and existing setups, such as Terraform, Ansible, or Kubernetes. The Terraform provider is ideal for those already using Terraform, while the Grafana Ansible collection suits Grafana Cloud users familiar with Ansible. Grizzly provides a Kubernetes-inspired YAML interface, and the Grafana Crossplane provider integrates with Kubernetes manifests for GitOps pipelines. Lastly, the Kubernetes Grafana Operator is a native solution for managing Grafana within Kubernetes environments. Each tool allows for different levels of resource management and integration, offering flexibility to users based on their existing infrastructure and preferences.

In December 2022, Grafana Labs announced the integration of Pyroscope, an open-source continuous profiling project, into their platform, under the new name Grafana Pyroscope. This integration has introduced flame graphs as a visualization tool for continuous profiling data within Grafana, enabling users to identify resource consumption patterns in their applications. Flame graphs visually represent function call stacks with hierarchical and color-coded blocks, helping users to pinpoint which functions consume the most resources and allowing them to drill down into specific areas for optimization. Additionally, Grafana provides a top table view that complements the flame graph by displaying profiling data in a tabular format, facilitating easier data analysis and search functionality. Grafana Labs continues to refine Grafana Phlare, encouraging user feedback and engagement for future improvements, while also promoting their Grafana Cloud service for comprehensive metrics and dashboard management.

Ricardo Liberato shares his experience in using Grafana, Prometheus, and Grafana Loki to monitor high cardinality jobs at a Fortune 100 life sciences company, highlighting the synergies between these tools for effective observability. By integrating Loki with Prometheus, the team was able to monitor job-run performance beyond infrastructure, enabling them to reduce load cycle times significantly and achieve 40% savings on cloud infrastructure costs. The implementation involved custom-built exporters and dashboards that aggregate data from various enterprise systems, providing insights into compute and memory efficiency and allowing for better resource allocation. This setup not only improved operational control but also provided feedback to development teams to enhance production performance. The combination of metrics and log information facilitated a deeper understanding of job performance and resource usage, leading to further cost optimizations through predictive scaling of resources.

Grafana Loki 2.7 introduces a significant experimental feature with the adoption of a Prometheus TSDB-inspired index, aimed at reducing storage footprint and enhancing query performance. Initially previewed at ObservabilityCON 2022, this update addresses challenges in handling high log volumes, improving query speeds, and reducing infrastructure costs. The TSDB index facilitates more efficient resource utilization, leading to a 75% reduction in index size and faster query processing, achieving up to 400GB/second scan speed. Users can activate the TSDB index by updating their configuration, though it remains in the experimental phase. The release also brings enhancements to Promtail, Azure blob storage support, and additional label_format fields, while community feedback and internal testing continue to refine its capabilities. For further details, the Grafana Loki 2.7 release notes and upgrade guide provide comprehensive insights, and the Grafana Cloud offers an accessible platform for engaging with metrics, logs, traces, and dashboards.

Grafana Tempo 2.0 introduces TraceQL, a pioneering query language designed to enhance trace analysis by making it easier to locate specific traces, thereby advancing distributed tracing technology. Tempo, an open-source and scalable backend for storing queries, integrates seamlessly with Grafana, Prometheus, and Grafana Loki, and supports various open-source tracing protocols like Jaeger and OpenTelemetry. The new TraceQL language allows for sophisticated trace querying, enabling users to diagnose and respond to root causes more efficiently by extracting and searching through traces interactively. This development mirrors the impact of LogQL on Grafana Loki, transforming it into a highly popular open-source project. TraceQL's data-type awareness offers advanced operations for handling various data types, and the introduction of a new storage format using Apache Parquet is expected to unlock further data utility. Grafana Tempo 2.0, previewed at ObservabilityCON 2022, promises to transform trace analysis with its innovative features, and the release is anticipated to be available shortly, offering a new search experience in Grafana 9.3.1.