January 2022 Summaries

Synthetic Monitoring in Grafana Cloud, a black box monitoring tool, provides insights into application performance and uptime by observing external requests from around the world, highlighting user experiences. While it effectively determines whether requests succeed or fail, its traditional limitations are addressed by the new traceroute feature, which traces the path of a request through the internet. This enhancement allows users to visualize network paths, detect changes over time, and identify the number of hops to a destination, offering deeper insights into network issues between clients and servers. Traceroute helps determine whether problems originate inside or outside the network, thus clarifying potential external issues if certain locations experience failures while others do not. The tool requires no special configuration and invites user feedback for further development, while Grafana Cloud offers easy access to metrics and monitoring with a free tier and various plans.

Grafana Labs, a company that has operated as a fully remote organization since its inception, is hosting its first virtual open house to showcase its remote-first culture and offer insights into working at the company. With a team spread across more than 50 countries and a workforce of over 600 employees, Grafana Labs emphasizes autonomy, trust, teamwork, and collaboration without geographical constraints. The virtual event aims to provide potential candidates and interested individuals with an overview of Grafana Labs' products, culture, and the unique aspects of being part of a remote-first company, including their focus on diversity, inclusivity, and effective communication across time zones. Attendees will have the opportunity to participate in general sessions and breakout sessions with senior leaders from various departments to learn about team structures, working styles, and company goals, as well as to explore open positions and the company’s approach to work-life integration.

In a brief tutorial video, Grafana Labs demonstrates how to build a Prometheus query using Grafana 8.3's Explore mode, which is accessible once Prometheus data sources are set up in Grafana. The video explains how users familiar with PromQL can directly input queries, while those new to it can benefit from an introduction and best practices guide. It highlights the functionality of the metrics browser, which assists users in discovering available metrics and constructing queries with specific metrics and labels. For those interested in further exploration, additional documentation is available, and Grafana Cloud offers a free tier to facilitate easy access to Grafana and Prometheus features.

Grafana, known for its compatibility with over 100 data sources, prominently supports Prometheus, a favored choice for those looking to expand their monitoring capabilities within Grafana's "big tent" philosophy. This brief tutorial video demonstrates how to quickly set up a Prometheus data source in Grafana, covering customization options such as authentication and scrape intervals within a concise timeframe. For more detailed guidance, Grafana offers comprehensive documentation and additional resources like the "Intro to metrics with Grafana" video. Grafana Cloud provides an accessible entry point with its free tier, offering a generous allowance of metrics, logs, and traces to help users get started with Grafana and Prometheus.

Grafana Tempo 1.3 introduces several new features, including the ability to search the backend datastore, which will soon be available in Grafana Cloud Traces. The update also includes improvements such as auto-forget compactors to address issues related to unhealthy states in the hash ring, better protection from large traces with configurable size limits, and minor breaking changes like the removal of the `Push` GRPC endpoint from ingesters and a port change for the OTel GRPC. The release aims to enhance operational efficiency by using serverless technologies to improve search speed and includes plans for future improvements in backend trace formats to support more efficient search and tag-based queries. Users are encouraged to join community calls and participate in forums for the latest updates and support, with the option to start using Tempo through Grafana Cloud, which offers a free tier for new users.

Ryan Perry discusses the performance enhancements made to Go applications using Pyroscope, particularly focusing on the transition from push to pull mode for profiling data. Pyroscope, initially facing performance issues in pull mode due to the server-side load of transcoding pprof profiles, required optimization strategies to manage the increased CPU demands. The initial optimization involved caching data in a hash map, which only marginally improved performance by trading a binary search for map lookups. A more effective solution eliminated the need for binary search by directly accessing objects in sorted arrays based on their IDs, significantly reducing CPU usage. Perry highlights the importance of continuous profiling for identifying and resolving inefficiencies in code, emphasizing that understanding the structure of pprof objects was key to optimizing performance. He invites readers to explore Pyroscope documentation for further insights into profiling and performance bottlenecks.

Owen Diehl's article explores the process of diagnosing and resolving performance issues in Grafana Loki's read path, focusing on a real-world investigation at Grafana Labs. The piece details how the author identified resource overconsumption by a single tenant, which led to congestion and elevated latencies across the system. This investigation involved leveraging Grafana dashboards, Prometheus, and Loki within a Kubernetes cluster to pinpoint the root cause, which was found to be expensive queries with long lookback intervals. As immediate mitigation, scaling the read path and reducing the offending tenant's query parallelism were suggested, along with recommending recording rules for recurring queries. For long-term solutions, Diehl emphasizes scaling the read path further, enhancing instrumentation for quicker issue identification, and implementing better quality of service controls to prevent resource overconsumption by a single tenant. The article offers a comprehensive look at using metrics and logs to troubleshoot and optimize performance in a multi-tenant environment, with the ultimate goal of ensuring a balanced resource allocation.

Pyroscope, a profiling tool acquired by Grafana Labs in 2023, has introduced an ad hoc profiling mode that complements its continuous profiling capabilities. This new feature is designed to provide a seamless and efficient way to profile scripts, offering users the ability to generate and share interactive flame graphs in a standalone HTML format. The ad hoc mode is beneficial for profiling scripts and analyzing performance issues without the need for continuous monitoring. With the ability to export and share profiles easily, users can leverage Pyroscope's visualization and UI features to better understand and communicate performance data. Additionally, Pyroscope stores profiles permanently, allowing for ongoing analysis and comparison. Future enhancements are planned, including support for more languages and formats, as well as improved UI functionalities to further enrich the profiling experience.

Continuous profiling for Golang applications, as demonstrated using Pyroscope, allows developers to monitor and debug performance issues effectively by tagging data in meaningful ways. The example simulates a ride-sharing app with endpoints for ordering bikes, cars, and scooters, running on servers across different regions. Pyroscope helps identify performance bottlenecks by tagging data based on static attributes like server region and dynamic attributes such as the vehicle type. By analyzing flame graphs generated from profiling data, developers can pinpoint resource-heavy functions, like the OrderCar function consuming significant CPU resources in the us-west-1 region due to the mutexLock function. The tool's comparison view enables comparing performance across different time periods, revealing varying CPU utilizations. Pyroscope's tagging capability has been utilized by companies for various purposes, such as tagging Kubernetes attributes and environments, enhancing the monitoring and debugging process. The integration of continuous profiling is becoming an essential component of observability, with ongoing efforts to improve Golang integrations and explore new features.

GopherCon UK 2021 was held in-person at the Brewery in London, requiring proof of vaccination or a negative COVID test for entry, and adhered to government guidelines. Hosted by Mat Ryer, the conference featured a variety of engaging talks, including Bryan Boreham's keynote on building large systems, Tiago Mendes' session on coordinating data changes in distributed systems, and Daniela Petruzalek's presentation on indie game development in Go. The event also covered topics like acceptance tests, Behavior-Driven Design (BDD), and test coverage in Go, with speakers such as Riya Dattani, Chris James, and Eleni Fragkiadaki providing insights. Additionally, Tom Camp discussed creating a chat app to learn new programming languages, while Alex Ellis shared the development journey of OpenFaaS. The conference concluded with acknowledgments to organizers, technical staff, and speakers for their contributions, and all sessions were made available on YouTube for broader access.

Network monitoring is crucial for organizations to prevent service disruptions and revenue loss due to network outages, which are increasingly common due to the complexity and size of modern networks. Grafana and Prometheus offer robust solutions for monitoring network performance, with Grafana providing visualization capabilities and Prometheus handling data collection and alerting. The Simple Network Management Protocol (SNMP) is a key method used to monitor network devices, and Grafana Labs offers an SNMP mixin that integrates with Prometheus, allowing users to configure dashboards and alerts based on specific network metrics. These tools enable network engineers to diagnose and resolve issues quickly by visualizing data such as CPU load, memory utilization, and bandwidth use, thereby enhancing proactive network management. The guide encourages users to explore Grafana Cloud for an accessible entry point into metrics, logs, and dashboards, offering a free tier and various plans to suit different needs.

Grafana has released versions 8.3.4 and 7.5.13, addressing a security vulnerability related to OAuth forwarding for data sources when API keys are used, which affects Grafana installations from version 7.5.x to 8.x. This issue, identified by Mikko Auvinen and classified as CVSS low, allows the OAuth Access Token of the most recently signed-in user to be used instead of the API token when forwarding is enabled, leading to unexpected behavior. Grafana recommends upgrading to these latest versions to resolve the issue, or alternatively, limiting the availability of API tokens if upgrading is not possible. The company acknowledges Auvinen's responsible disclosure and encourages reporting any security vulnerabilities via their dedicated email, with encryption preferred. Security announcements and mitigations are regularly updated on their blog and can be accessed through an RSS feed.

Grafana Labs hosted its second company-wide hackathon in December, following the success of the first event in June, with increased participation from 148 staff members across various roles. The week-long virtual event featured cross-functional teams that developed a wide array of projects, including innovations involving drones and race cars, as well as enhancements to Grafana's core products. The hackathon emphasized technical innovation, design, commercial potential, and community impact, and despite last-minute planning, the quality of submissions impressed the judges. As a result, six finalist presentations were showcased for a company-wide vote instead of the planned five, with Grafana Labs CEO Raj Dutt and VP of Corporate Development Ash Mazhari expressing admiration for the creativity and rapid innovation demonstrated by participants. Many of the presented ideas are set to be further developed and could be integrated into future product roadmaps, highlighting the hackathon's role in fostering both personal and company growth.

Grafana Labs successfully executed its third virtual Grafana Cloud offsite in 2021, bringing together over 150 employees from 32 countries, spanning various time zones and both fall and spring hemispheres. The event aimed to foster a shared vision and deepen social bonds among team members, offering a blend of leadership talks, technical deep dives, and engaging social activities like breakout sessions and lightning talks. Feedback from previous events led to a well-received three-day format, balancing content delivery and participant engagement. Despite challenges like the absence of in-person interactions due to the pandemic, the offsite was deemed a success, with high satisfaction ratings from attendees. Grafana Labs utilized familiar tools such as Slack and Zoom to facilitate seamless participation and interaction, creating a sense of community and connection. Looking forward, the company acknowledges the benefits of both virtual and in-person events, with plans to continue incorporating virtual elements to accommodate diverse needs and reduce travel impacts.

In the blog post, the author, Anders Østhus, a DevOps Engineer at Proactima AS, discusses the integration of Grafana Tempo and Linkerd for distributed tracing. The article highlights the benefits of distributed tracing, such as providing insights into request flows, latency visibility, and troubleshooting tools. Østhus shares his experience using Grafana Tempo, a trace-neutral backend that supports various trace formats and is compatible with different environments. The post details the configuration process for integrating Grafana Tempo with a stack comprising Linkerd, NGINX Ingress Controller, and Grafana, emphasizing the use of the B3 Propagation standard for Linkerd. The guide also touches on optional integration with Grafana Loki for enhanced log-to-trace capabilities. The author encourages readers to explore this setup for better data visibility and management, offering steps for testing and verifying the configuration, and promoting further learning opportunities through webinars and Grafana Cloud's free tier offerings.

Grafana Cloud provides guidelines for creating effective alerts on synthetic monitoring metrics, utilizing predefined alerts and PromQL expressions for various monitoring needs such as uptime, reachability, average latency, and error rates. The platform offers a predefined alert on the probe_all_success_sum metric, while users can define alert thresholds for specific metrics like reachability and error rates, both generally and by individual probes. Alerts are built using PromQL expressions that evaluate data over a specified time and can transition from pending to firing states if conditions persist, with users encouraged to test and tweak thresholds to ensure accuracy. Grafana Cloud also offers documentation and resources for further learning, along with a free account option to explore its features.

TomTom, a leader in location technology, has significantly enhanced its developer experience by integrating Grafana Cloud, as shared by Site Reliability Engineer Carl Meert at ObservabilityCon 2021. The implementation has facilitated a reduction in Mean Time to Recovery (MTTR) and improved service visibility, addressing previous challenges of disparate monitoring tools and unshareable data across teams. By consolidating various business metrics into Grafana dashboards, TomTom achieved a comprehensive end-to-end view of their services, aiding in incident management and SLA tracking. This visibility has been further enhanced by introducing weekly technical operational reviews that promote collaboration and knowledge sharing among engineers. Additionally, the team has embraced synthetic monitoring to ensure full visibility across their services, and they are actively exploring new functionalities such as distributed tracing with Grafana Tempo. The transition to Grafana Cloud has not only fostered an observability-centric culture but also opened avenues for future advancements in incident response and data integration.

Vardan Torosyan, an engineering manager at Grafana Labs, reflects on his experience building a remote-first team during the pandemic, highlighting both challenges and successes in adapting to a rapidly growing company. After joining Grafana Labs in December 2020, he navigated the complexities of onboarding during a global health crisis while integrating into a team that doubled in size to over 300 employees. Torosyan describes the strategic formation of the Grafana Enterprise Operations team, which involved seeding it with existing team members and focusing on delivering high-value capabilities to enterprise customers. The onboarding process was meticulously designed to ensure new hires felt welcomed and equipped to contribute, with personalized plans and support from mentors. A virtual retreat provided an opportunity to reflect on team dynamics and reinforce a cohesive remote-first culture. Looking ahead, the team aims to deepen customer and community engagement while continuing to expand, with a focus on delivering impactful solutions.

The k6 Cloud app plugin for Grafana facilitates the correlation of QA data and system metrics within Grafana dashboards, enhancing performance testing and system behavior analysis. Initially released as a beta feature following Grafana Labs' acquisition of k6, the plugin allows users to manage and visualize cloud tests, correlating testing metrics with other system metrics to detect root causes. It supports installation on both Grafana Cloud and local instances, requiring an API token for configuration. Users can navigate tests, re-run cloud tests, and easily integrate testing results into existing dashboards. The app offers enhanced visualizations consistent with k6 Cloud, although it currently supports fewer features. Future updates aim to improve visualization and integration capabilities, with feedback encouraged via the k6 #grafana Slack channel. Available for k6 Cloud, Grafana Cloud, Grafana Enterprise, and self-hosted instances, the app underscores the collaborative effort between Grafana and k6 to advance performance testing solutions.

Grafana Labs is hosting webinars in January featuring their new Sentry plugin and distributed tracing tools, aiming to enhance observability in development processes. On January 13, a session led by Grafana Labs Product Manager Jay Goodson and Sentry Senior Software Engineer Nisanthan Nanthakumar will demonstrate how the Sentry plugin can consolidate key business metrics into a unified Grafana dashboard, providing a live demo and interactive Q&A. Another webinar on January 27, presented by Grafana Labs’ Product Manager Jen Villa, will cover distributed tracing using Grafana Tempo, an open-source tool for handling distributed traces, and introduce the features of Grafana Enterprise Traces (GET) for secure, scalable tracing in enterprise settings. These webinars are designed to equip participants with practical skills and insights to improve their observability strategies.

Civo, a Kubernetes service provider, faced challenges with legacy logging solutions in managing their complex infrastructure, prompting them to adopt Grafana Loki for distributed logging at scale. During ObservabilityCon 2021, Civo's Site Reliability Engineer Anaïs Urlichs and Principal Engineer Alex Jones demonstrated how they use Promtail with Grafana Loki and Grafana dashboards to collect, aggregate, and visualize logs, enhancing their observability stack. They shared practical tips for optimizing Grafana Loki, such as configuring it to match infrastructure needs, using labels for efficient log retrieval, and leveraging LogQL for advanced querying. The session also highlighted plans for future enhancements such as inter-regional logging and synthetic monitoring, and offered resources for those interested in exploring Grafana Cloud for scalable Kubernetes logging solutions.