October 2022 Summaries

ObservabilityCON, hosted by Grafana Labs, is set to take place in New York City on November 2, with the option for virtual attendance. Although the in-person event, including pre-conference workshops, is sold out, attendees can register to watch the keynote online. The event will be held at etc.venues on Madison Avenue, with registration starting at 8:30 a.m. ET and activities concluding at 6 p.m. ET. The day will feature a keynote, nine breakout sessions on observability, and an expo with technical experts available for questions. Notable speakers from companies such as Adobe, JPMorgan Chase, and Wells Fargo will present, and the day will end with a networking reception. Attendees must bring proof of COVID-19 vaccination and a government-issued ID, while masks are encouraged. Additionally, there are opportunities for online engagement through Slack and social media using the hashtag #ObservabilityCON. Those unable to attend in person can still participate via the livestream, which will feature announcements from Grafana Labs executives.

GrafanaLive events, organized by Grafana Labs, marked a return to in-person gatherings for the Grafana community, occurring in various global locations such as Chicago, London, and São Paulo, with upcoming events planned in Australia and India. These events served as a platform for Grafana users, community members, and observability experts from companies like Adobe and DRW Trading Group to share insights and updates on the Grafana Labs LGTM stack, which includes Loki, Grafana, Tempo, and Mimir for different observability functions. The events featured product demos, breakout sessions on topics like log management with Grafana Loki, and networking opportunities, all while supporting the Girls Who Code initiative through ticket proceeds. Participants were able to engage in technical discussions at the "Ask the Experts" booth and learn about community-driven use cases, highlighted by the release of Grafana 9.0. Grafana Labs plans to continue fostering community connections through future events and online platforms like ObservabilityCON 2022.

Optum, a U.S.-based health services company, has leveraged Grafana Enterprise to improve its website's performance monitoring system, aiming to reduce user stress during significant life events by providing a fast and reliable web experience. By uniting disparate data sources, Optum created a robust architecture that offers real-time insights through user-friendly Grafana dashboards, accessible to all stakeholders from executives to development teams. This system helps track metrics such as site performance, load times, and errors, allowing for quick identification and resolution of issues, which has led to reduced mean time to repair (MTTR) and improved customer satisfaction. The dashboards are designed to provide both high-level overviews and detailed drilldowns, enabling efficient decision-making and continuous enhancements to the user experience. Despite challenges with internal standardization and complex component interactions, Optum successfully visualized performance data, facilitating improvements in member experience and operational efficiency without the need to re-implement their entire system.

Kubernetes node observability is a crucial component of maintaining the health and efficiency of Kubernetes infrastructure, acting as the backbone for application performance. Grafana Cloud's Kubernetes Monitoring solution offers a robust toolset for monitoring node health and resource usage, addressing key challenges such as identifying potential issues and optimizing resource management. The solution provides features that display node conditions and resource usage with color-coded indicators, helping reduce the mean time to resolve issues and enhancing resource management by mapping pods to nodes based on utilization. This approach not only supports high availability and optimal node configuration but also offers insights into the health and status of all pods within a node, empowering users to take data-driven actions for improved infrastructure management. The tool is designed to ensure that Kubernetes clusters perform efficiently and remain resilient, making it a valuable asset for organizations using Grafana Cloud for Kubernetes monitoring.

Grafana Labs and Isovalent have formed a strategic partnership to enhance observability for Kubernetes and cloud-native infrastructure by combining Grafana's open-source observability stack with Isovalent's Cilium, an eBPF-powered connectivity and security tool. This collaboration addresses challenges in gaining comprehensive connectivity observability for modern API-driven applications, which involve complex, layered network structures and ephemeral identifiers, by leveraging eBPF technology to provide deep, service-identity-aware insights without requiring changes to application code. Cilium generates detailed connectivity metrics and events that integrate seamlessly with Grafana's LGTM stack, offering powerful solutions for issues such as API-layer faults and network-layer bottlenecks. The partnership is seen as a significant advancement in simplifying and enhancing the monitoring and troubleshooting capabilities for application owners and Kubernetes platform teams, with ongoing developments expected to further expand its impact.

KubeCon + CloudNativeCon North America 2022 in Detroit is a key event for the cloud native community, featuring a variety of sessions led by Grafana Labs experts. Highlights include discussions on transitioning Jaeger to OpenTelemetry for enhanced application tracing, with insights from Grafana's Joe Elliott and Logz.io's Jonah Kowall. Prometheus, a standard for cloud native metrics, will be explored by Ganesh Vernekar and Richard Hartmann, offering both introductory and advanced insights. Nadine Vehling and Matthias Loibl will present Pyrra, a tool for managing Prometheus service level objectives (SLOs) in Kubernetes, with practical demonstrations of setup and alerting features. Éamon Ryan and Heds Simons will showcase how Grafana Labs utilizes GitOps in Kubernetes environments to streamline operations without requiring users to interact with Git directly. The conference provides opportunities to learn about best practices in cloud native technologies and engage with experts in the field.

As enterprises increasingly adopt cloud-native architectures, the rapid growth of metrics cardinality in observability systems like Prometheus has become a significant concern due to its impact on infrastructure costs and performance. This growth is driven by the complexity of microservices and Kubernetes environments, where each layer and component generates unique time series data, compounded by the ease of instrumentation and lack of centralized governance. High cardinality leads to increased infrastructure demands and can slow down query performance, affecting troubleshooting efficiency. To address this, enterprises are advised to gain visibility into which metrics and labels contribute most to cardinality, identify valuable metrics, and optimize data collection practices, such as adjusting scrape intervals and refining label usage. Tools like Grafana Cloud offer features for tracking and managing metrics cardinality, helping organizations attribute costs to specific teams and optimize their observability strategies effectively.

Grafana Labs has enhanced the scalability and efficiency of its log aggregation system, Grafana Loki, by implementing the Kubernetes-based Event Driven Autoscaler (KEDA) to manage the fluctuating demands of querier components within Grafana Cloud Logs. This move addresses previous challenges of manual scaling, such as delays, over-provisioning, and operational toil, by automating the scaling process based on specific metrics like the number of inflight requests. KEDA, which integrates with Prometheus metrics and is built on top of Kubernetes' Horizontal Pod Autoscaler (HPA), allows for more responsive and cost-effective scaling decisions. By fine-tuning thresholds and stabilization windows, Grafana Labs achieved smoother scaling transitions, reduced the average number of replicas, and maintained query latency without degradation. This approach not only lowers operational costs but also enhances the reliability and performance of Grafana Loki, making it more user-friendly and cost-efficient for various deployment scales.

Varland Plating, a long-standing electroplating company in Cincinnati, Ohio, modernized its industrial IoT monitoring by transitioning from a limited legacy system to a more efficient Telegraf-InfluxDB-Grafana (TIG) stack. This shift, shared by VP of Technology Toby Varland during GrafanaCONline, enabled the company to improve data visualization and analysis, essential for tracking metrics such as temperature and electrical current during the electroplating process. Grafana's ease of use and minimal setup requirements made it an appealing choice, allowing for customizable and comprehensive visualizations that enhance decision-making. The new system supports multiple remote connections and offers robust data aggregation capabilities, which have been instrumental in optimizing operations and providing valuable insights despite the company's budget constraints. The transformation was a significant technological upgrade, allowing Varland Plating to better monitor and utilize its data for improved business outcomes.

Grafana 9.2 introduces a new panel help feature aimed at improving the troubleshooting process for dashboard editors by allowing them to better communicate issues with Grafana Labs support teams. This feature enables users to send a panel’s JSON model, including query response data with randomized sensitive information and all visualization settings, directly to support, thereby reducing friction and aiding in issue reproduction. Users can access this feature via the Panel menu, where they can create and send snapshots as GitHub issues or through the Grafana help system, which helps the support team replicate the dashboard environment for effective problem-solving. Additionally, the release encourages users to explore other new features, participate in community discussions, and utilize resources such as webinars and documentation to enhance their Grafana experience.

Nicolas Ventura, a critical facilities engineer at NERSC, discusses the implementation of Psychart, a tool for integrating psychrometric charts with Grafana to monitor the health of high-performance computing (HPC) systems by visualizing air conditions. NERSC, operated by Lawrence Berkeley National Laboratory, uses this tool to ensure optimal conditions for its HPC systems by tracking temperature, humidity, and flow rates through sensors. Psychart, developed using Grafana and Psychrolib, allows engineers to visualize critical environmental data, helping maintain system efficiency and prolong equipment life by adhering to ASHRAE guidelines. This tool, which gained traction after being published in May 2022 and featured in various showcases, enables operators to quickly assess air states and make necessary adjustments, enhancing the facility's operational efficiency. Additionally, Psychart's flexibility allows it to be applied beyond data centers to other fields such as chemical processing and atmospheric monitoring, with its open-source nature contributing to its widespread adoption.

Kubernetes monitoring is crucial for managing containerized applications, offering benefits such as real-time alerts for early error detection, enhanced workload management, easier troubleshooting, real-time cost visibility, and powerful insights into application usage. Effective monitoring can prevent service disruptions, optimize resource use, and provide actionable data that informs future product decisions. Grafana Cloud offers a comprehensive Kubernetes Monitoring solution that simplifies the process by automating tasks, thus minimizing errors and allowing users to focus on critical aspects of their clusters. With prebuilt dashboards and alerts, Grafana Cloud's Kubernetes Monitoring provides immediate access to metrics, logs, and events, making it easier for SREs, developers, and DevOps administrators to maintain and optimize their systems. James Walker, a seasoned software developer and technical writer, highlights the importance of these tools in maintaining robust and cost-effective Kubernetes deployments.

On October 11, Grafana released version 9.2.0, which includes new features, bug fixes, and critical security patches for vulnerabilities identified as CVE-2022-39229, CVE-2022-39201, CVE-2022-31130, and CVE-2022-31123. These security issues were discovered as part of internal audits and external research, affecting various versions of Grafana, and involve problems such as improper authentication, data source and plugin proxy endpoint vulnerabilities, and plugin signature bypass. Grafana Labs has released corresponding security patches for versions 9.1.8 and 8.5.14, and has coordinated with cloud providers like Amazon and Azure to ensure their managed services remain secure. The recommended action for users is to upgrade their Grafana instances to the latest versions where patches have been applied, and further security communications are facilitated through Grafana's blog and the option for users to report vulnerabilities via email, using PGP encryption for confidentiality.

Grafana 9.2 introduces a variety of updates and new features designed to enhance dashboard creation and user support, including a new panel help option that streamlines issue reporting and troubleshooting through a wizard that generates data snapshots for support teams. The release also debuts the Canvas panel in beta, allowing users to design custom visualizations with static and dynamic layouts, and simplifies the query variable editor for Grafana Loki to ease variable creation. Improvements to the Google Cloud Monitoring UI provide a more intuitive query-building experience, while transformations now support both INNER and OUTER JOINS for clearer data visualization. Alert rules have been updated to default to an Error state on execution failure, and Grafana 9.2 enhances authentication and authorization by allowing role assignment through Terraform and improved single sign-on capabilities. The update also includes UI enhancements like a role picker for team creation and the ability to organize custom roles into groups, facilitating efficient role management.

Landing a junior engineering role at Grafana Labs involves a multifaceted approach, focusing on candidates' technical skills, soft skills, and overall enthusiasm for learning and problem-solving. Grafana Labs values a candidate's ability to communicate, collaborate, and work effectively in a remote-first environment, as well as their eagerness to grow and tackle complex solutions. The company distinguishes between entry-level and junior engineers based on experience, with entry-level positions providing a pathway to junior status through mentorship. The hiring process is entirely remote, comprising stages like a recruiter screen, a hiring manager interview, and a skills assessment. Grafana Labs also emphasizes the importance of open source involvement and offers various opportunities for growth and contribution within its global team. For those seeking to begin their engineering career, Grafana Labs encourages exploring opportunities at diverse companies, practicing programming skills, networking, and attending career fairs, while also inviting interested candidates to explore their careers page for potential openings.

In the year following Grafana Labs' acquisition of k6, the integration has been marked by a strategic focus on maintaining team autonomy while fostering collaboration between the two entities. Initially inspired by the potential synergies between k6's performance testing capabilities and Grafana's observability tools, the acquisition emphasized keeping k6 as an independent entity to retain its core team and culture. This approach proved successful, as most k6 employees remained post-acquisition, and the team size nearly doubled. Over time, efforts were made to integrate k6 into Grafana Labs' broader operations, particularly in sales, product development, and cross-team collaborations, which enhanced the overall product offering and company culture. The acquisition not only strengthened Grafana Labs' position in observability but also set a precedent for future acquisitions by prioritizing people, product integration, and cultural evolution, ensuring a cohesive and unified company vision.

Grafana Cloud Link is an experimental feature that enables users to create a seamless hybrid environment between local Grafana instances and Grafana Cloud, offering the flexibility to manage and monitor data across both platforms. Developed from a hackathon project, this gateway allows users to collect metrics and log data from local workloads and transfer them to Grafana Cloud, or to monitor Kubernetes clusters via the cloud service. It also provides the capability to push local data sources and dashboards to Grafana Cloud, helping users explore the benefits of a managed instance without fully migrating. Grafana Cloud Link simplifies the process of integrating local and cloud environments through its Import and Export functions, making it easy to test and interact with cloud solutions using existing local setups. Currently available for early access by request, the feature encourages user feedback to enhance the overall Grafana experience.

Green City Solutions, co-founded by Liang Wu, is a biotech startup in Bestensee, Germany, that tackles urban air pollution by integrating moss with IoT technology to create biofilters. These biofilters utilize the natural air-purifying properties of moss, which binds and metabolizes air pollutants, and are optimized through sensors and IoT to maintain ideal growth and filtration conditions. Grafana dashboards play a crucial role in visualizing and managing data from these biofilters across various European cities, enabling real-time monitoring and analysis of environmental conditions and system performance. This technology not only supports the efficient management of the company's moss farm near Berlin but also provides customers, including major companies and cities, with insights into air quality improvements and pollution reduction. Green City Solutions aims to significantly impact air pollution by removing substantial amounts of carbon dioxide and other pollutants from the atmosphere by 2030.

Grafana Cloud has introduced a new SNMP integration that simplifies the monitoring of network devices such as routers, switches, and servers by leveraging the Simple Network Management Protocol (SNMP), which operates primarily on UDP ports 161 and 162. This integration is accessible to all Grafana Cloud users, including those on the free tier, and uses an embedded SNMP exporter within the Grafana Agent to automatically collect data from configured SNMP targets. Users can set up an account, install the integration, and configure the Grafana Agent to begin monitoring their devices. The integration comes with a prebuilt dashboard offering comprehensive statistics and metrics, such as uptime, network traffic activity, and interface statuses, providing an overview of all network devices. Additionally, SNMP monitoring can be extended to Linux hosts by installing the snmpd and configuring the Grafana Agent. Detailed setup instructions and further information are available in the SNMP integration documentation, and the community is encouraged to provide feedback via Grafana Labs Community Slack.