July 2022 Summaries

Laura Benz's transition to Grafana Labs is recounted through her first 30 days at the company, highlighting her pivot from a career in aircraft maintenance to technology via a coding bootcamp. Her initial weeks at Grafana Labs emphasized the company's culture of transparency and community engagement, where she appreciated the open communication channels and public access to information such as Objectives and Key Results (OKRs). Benz's onboarding experience was thorough, featuring a personalized plan and an onboarding buddy, which facilitated a smooth transition into her role in frontend development. The company-wide GrafanaFest in Whistler offered a unique opportunity for in-person networking and understanding the organizational structure. Benz's narrative underscores the supportive environment at Grafana, encouraging social interaction and learning from colleagues, while she anticipates deepening her technical expertise and personal projects utilizing Grafana's tools.

Pipedrive, a company known for its CRM platform, faced challenges with the scalability of their Prometheus-based observability setup as their infrastructure expanded across multiple data centers and AWS regions. The limitations of Prometheus became evident when the system began to crash and run out of memory under high load, leading to delays and lost observability data. To address these issues, Pipedrive implemented Grafana Mimir, which offered enhanced scalability and performance improvements, such as fast query performance and per-tenant limits to manage data ingestion and queries more effectively. The migration to Mimir was seamless and transparent for users, and it allowed Pipedrive to handle between 12 and 15 million active series while fine-tuning high cardinality metrics. The transition reduced downtime and false alerts, improved overall system stability, and set the stage for further optimization and expansion of their observability stack. As a result, Grafana Mimir has become a cornerstone of Pipedrive's monitoring strategy, with plans to explore additional Grafana tools like Loki and Tempo in the future.

Grafana Loki 2.6 introduces significant features such as cross-tenant query federation and targeted log line deletion, enhancing its capabilities for multi-tenant data management and privacy compliance. The cross-tenant query federation enables users to query multiple tenants at once, consolidating results into a single output, which simplifies workflows and reduces operational overhead by allowing data access without physically separating databases for different teams or business units. This feature is particularly useful for administrators requiring a global view of logs across tenants. Additionally, targeted log line deletion addresses the need for precise removal of sensitive information, such as personally identifiable information (PII) or data requested for deletion under regulations like GDPR. Previously, deleting logs involved less precise methods, but now users can utilize LogQL queries to specifically identify and remove unwanted log lines, with immediate frontend filtering making the deletion appear instantaneous. In Grafana Enterprise Logs (GEL), these features are integrated with access control measures to ensure security and proper permissions for executing cross-tenant queries and deletions. These updates represent a significant step forward in data management efficiency and compliance within Grafana Loki's infrastructure.

Kubernetes Monitoring in Grafana Cloud offers an integrated solution for managing Kubernetes environments by addressing gaps in logging capabilities and enhancing the observability of clusters. This service utilizes Grafana Loki for log aggregation, compatible with Prometheus, allowing seamless correlation between metrics and logs to diagnose infrastructure issues more effectively. Users can access a range of features, including pre-built dashboards and alerting rules, through the Grafana Cloud platform, which is available in a generous free tier. By deploying the Grafana Agent and integrating it with Kubernetes infrastructure, users can collect and visualize pod logs alongside metrics, facilitating quicker identification of root causes. The platform's cluster navigation view further aids users in detecting issues by providing a comprehensive, nested interface for correlating metrics and logs.

Grafana Labs has introduced Grafana Mimir, an open-source time series database designed for high scalability and performance, capable of managing over 1 billion active series. Initially supporting Prometheus metrics, Mimir now extends its compatibility by including experimental write proxies that allow the ingestion of metrics from Graphite, Datadog, and InfluxDB without requiring users to alter their existing code. This initiative aligns with Grafana’s philosophy of enabling seamless data ingestion from multiple sources, similar to how Grafana serves as a centralized data visualization tool. The new proxies are available on GitHub and facilitate the translation of various metrics into Prometheus time series, supporting Grafana Cloud and local installations. As part of ongoing development, Grafana Labs plans to consolidate these proxies and enhance features like logging, tracing, and maintainability, aiming to establish Mimir as the ultimate tool for storing all metrics.

Grafana Labs, founded in 2014 by Raj Dutt, Anthony Woods, and Torkel Ödegaard, has seen rapid growth, establishing itself as a leading open-source company with a significant suite of observability tools, including Grafana Loki, Mimir, Tempo, and OnCall. The company emphasizes a culture of transparency, autonomy, and accountability, aiming to maintain these values while scaling. The co-founders reflect on the challenges and successes of growing the company during the pandemic, underscoring the importance of relationships and maintaining company culture amidst rapid expansion. They discuss the evolving definition of observability, the role of open source in commoditizing observability, and the importance of balancing high performance with avoiding burnout. The company is committed to fostering a supportive environment, emphasizing the need for effective hiring practices and onboarding processes to sustain its culture as it continues to grow.

Dish Network has significantly enhanced its satellite communications monitoring by implementing Grafana dashboards and Grafana Loki to improve uptime and system efficiency. Systems Administrator Engineer Ted Raymond explained during GrafanaCONline 2022 how these tools have transformed their operations, allowing real-time monitoring of equipment, antennas, and weather conditions, which is crucial for maintaining service for their 20 million U.S. customers. The team has moved from inefficient manual data collection to automated real-time data access, enabling swift problem resolution and collaboration across teams. By using Grafana Alerting, they can address issues like antenna performance before they affect uptime, and with Grafana Loki, they efficiently read log files for better server health monitoring. This modernization has not only closed the 1% uptime gap but also democratized data access, allowing users without programming expertise to build and interpret custom dashboards. The new system has enabled Dish Network to efficiently manage specific missions, such as de-orbiting spacecraft, by integrating various data sources into a single, comprehensive dashboard, ultimately fostering team unity and operational success.

Grafana Mimir, an open-source time series database for Prometheus, has achieved up to a 10x improvement in query performance through the implementation of two sharding techniques: time splitting and query sharding. These techniques allow Mimir to execute a single query across multiple CPU cores and machines in parallel, addressing the limitations of the Prometheus PromQL engine, which runs on a single thread. Time splitting divides a query by time intervals to distribute the load across multiple queriers, while query sharding distributes a query across multiple queriers based on the series subset, facilitating parallel execution. This method significantly reduces execution time for high cardinality and CPU-intensive queries, as evidenced by a reduction in execution time from 38 seconds to 4 seconds in one instance. Approximately 60% of customer queries on Grafana Cloud are sharded, demonstrating notable performance benefits. Grafana Labs continues to enhance query performance and expand sharding support for more query types, with further resources available through their documentation and webinars.

GrafanaLive is a series of in-person mini-conferences organized by Grafana Labs, aimed at bringing together the observability community to explore open-source technology. Starting on July 26, 2022, the events will be held in cities including the Bay Area, Washington, D.C., Chicago, and London, offering participants the chance to engage in hands-on demonstrations of Grafana Labs’ LGTM stack, which includes Loki for logs, Grafana for visualization, Tempo for traces, and Mimir for metrics. Attendees can also visit an "Ask the Experts" booth for personalized advice and listen to presentations from observability specialists from companies like Adobe and DRW Trading Group. The $75 registration fee will benefit Girls Who Code, a non-profit focused on closing the gender gap in tech. Additionally, free networking events in cities such as Boston, Dallas, Los Angeles, and Seattle will provide further opportunities for face-to-face interactions with industry professionals.

Grafana 9 introduced a new Prometheus query builder that simplifies writing PromQL queries by offering a visual interface, addressing the complexity of raw PromQL syntax. Initially developed as a Grafana Labs hackathon project, this feature allows users to build queries using a more intuitive approach with a dropdown menu for selecting metrics and labels, and a pipeline layout for operations. Users can also toggle between Code, Builder, and Explain modes for different levels of query interaction and understanding. This innovation aligns with Grafana’s goal of enhancing user experience across various data sources, with potential applications beyond Prometheus, such as in Grafana Loki. The development emphasizes collaboration and community feedback, with opportunities for users to contribute ideas and improvements through GitHub and Grafana Labs Community Slack. Additionally, Grafana Cloud offers an easy entry point for users to explore these new features with a free tier available for sign-up.

Apeel Sciences, an ag-tech company, is tackling the global food waste issue and its associated greenhouse gas emissions through the use of their innovative edible, plant-derived coating that extends the freshness of fruits and vegetables. By utilizing real-time Grafana dashboards and alerts, Apeel optimizes its treatment processes and manages global operations efficiently. The company employs sensors at its facilities worldwide to monitor key performance indicators like dryer temperature and flow rate, visualizing this data through Grafana dashboards for quick issue identification. Moreover, Grafana Alerting integrated with Slack notifies team members of anomalies in real-time, allowing for immediate response and troubleshooting. This system also allows Apeel to track throughput, helping them understand production rates and improve efficiency. The ability to securely share these insights globally through Grafana's SSO and access control features facilitates collaboration among teams, enhancing their efforts to significantly reduce food waste and its environmental impact.

Grafana has released versions 9.0.3, 8.5.9, 8.4.10, and 8.3.10 to address two high severity security vulnerabilities, CVE-2022-31097 and CVE-2022-31107, affecting versions from 5.3 to 9.0.1. The first vulnerability, disclosed by Maxim Misharin, is a stored XSS issue in Grafana Alerting, rated with a CVSS score of 7.3, for which upgrading or disabling Grafana Alerting is advised. The second, reported by the HTTPVoid team, is an OAuth account takeover vulnerability, rated at CVSS 7.1, mitigatable by disabling OAuth login. Grafana Labs has coordinated with cloud providers like Amazon Managed Grafana to ensure security patches are applied, and they encourage the responsible disclosure of vulnerabilities through their security contact. Security announcements and details on remediation are regularly posted on their blog.

Kubernetes Monitoring in Grafana Cloud provides a comprehensive solution for tracking and managing Kubernetes clusters, offering both seasoned developers and novices a range of tools to enhance observability. By installing the Grafana Agent, users can quickly begin sending Kube-state metrics to Prometheus and Grafana, gaining immediate access to metrics, logs, and events via prebuilt dashboards and alerts. A notable feature is the cluster navigation view, which allows users to visualize and drill down into Kubernetes objects at every level, from namespaces to individual pods, to monitor CPU usage, pod status, and other critical metrics. The solution is available to all Grafana Cloud users, including those on the free tier, and integrates seamlessly with existing Kubernetes integrations, further streamlining the monitoring process. Additionally, Grafana Labs encourages feedback from the community as they continue to enhance their offerings, with plans to support out-of-the-box trace configurations for Grafana Tempo in the future.

Accessing and querying REST APIs with the Sqlyze plugin in Grafana involves using SQL to treat APIs as database tables, allowing for integration with various data sources, including NoSQL and document databases. The process requires installing an ODBC driver, such as the CData API driver, and setting up a data source in Grafana with necessary paths and API keys. The example provided uses ClickUp, a productivity tool, to demonstrate querying hierarchical data structures by creating template variables for different levels like Team, Space, and List, which are then used to filter and display tasks in Grafana dashboards. This method allows for creating interactive dashboards that integrate API data seamlessly, with potential applications beyond ClickUp, including tools like Jira, available for Grafana Cloud or Enterprise license users. The Sqlyze Enterprise plugin thus facilitates versatile data querying capabilities in Grafana, supporting SQL, NoSQL, document databases, and REST APIs.

Grafana 9 introduces significant updates aimed at enhancing navigation and search functionalities to improve user experience and support community observability efforts. Key features include a revamped vertical navigation bar, the ability to save favorite visualizations in a starred dashboards section, and the option to create panels and dashboards directly from the Explore module. Search capabilities have been advanced with the addition of panel title searches and dashboard previews, both of which are currently in beta and can be accessed by enabling specific feature flags. These enhancements, alongside the launch of the Grafana Loki query builder and a new command palette, were unveiled at GrafanaCONline 2022, with feedback actively solicited from the community to further refine the platform's usability.

Frederic Branczyk, founder and CEO of Polar Signals, has developed an open-source continuous profiling tool called Parca, which is designed to optimize resource usage and reduce latency without needing configuration changes. Inspired by Google's approach to profiling that led to significant infrastructure cost savings, Branczyk transitioned from working on Prometheus to create Parca, aiming to fill a gap in the market for continuous profiling tools. Parca uses techniques like sampling profiling and eBPF to minimize overhead while continuously analyzing resource usage, offering users insights that can enhance performance and efficiency. Building an open-source community around Parca, Branczyk emphasizes automation to reduce manual toil and focuses on creating value for users. By engaging with customers, especially in sectors like e-commerce, Polar Signals has gained valuable feedback, highlighting the importance of latency optimization over cost-saving, and continues to iterate on its solutions, aiming to offer continuous profiling as a service.

Dan Mihai Dinu details his experience configuring Grafana Loki, an open-source log aggregation system, with a Node.js e-commerce application he built for his father's honey business in Romania. To enhance the website's functionality and customer experience, Dinu integrates logging capabilities using the unofficial winston-loki client to send logs from Google's Firebase to Grafana Cloud, overcoming initial authentication challenges by applying a patch-package fix. He implements two Express middleware functions for logging incoming requests, response times, and unhandled errors, enabling detailed log analysis through Grafana's LogQL within Grafana Explore. Dinu builds dashboards to visualize metrics, which help identify and resolve a critical bug affecting customer orders, significantly improving the business's operational efficiency and preventing future sales losses. Through this process, he demonstrates the utility of Grafana Cloud's tools for monitoring and enhancing web application performance, highlighting the platform's capacity for both beginners and advanced users to gain insights and improve their applications.

Edgardo Peregrino, a freelance developer and robotics enthusiast, shares his journey of integrating robotics projects with cloud-native tools like Grafana, Prometheus, and Jaeger for real-time monitoring. Starting with a Raspberry Pi and progressing through various control methods for his robots, Linus and Torvalds, he expanded his skills by learning about cloud-native technologies in a scholarship program. Peregrino leveraged these tools to create a comprehensive monitoring system using the Flask Python library, Docker, and Kubernetes, enabling him to track robot performance with a Grafana dashboard displaying metrics like response time and processor utilization. His work demonstrates the potential of combining open-source tools with robotics to improve monitoring and diagnostics, with future plans to utilize a Raspberry Pi cluster for even greater innovation.

Grafana 9 introduced a new query builder for Grafana Loki at GrafanaCONline 2022, aiming to simplify the creation and editing of LogQL queries, especially for new users. The tool allows users to easily select log streams through a label filter and enhance queries with operations such as line filters, parsers, and aggregations. It offers features like tooltips for better understanding of operations, an "Explain" mode for a comprehensive breakdown of query components, and a visual interface for editing queries. Users can switch between text, builder, and explain modes without losing changes, making it easier to learn and refine queries. The Grafana team is actively seeking user feedback to further enhance the query builder, and they encourage participation in community discussions and user research. Users interested in exploring these new features can watch the "Deep dive into Grafana 9" session from GrafanaCONline 2022 and engage with the Grafana community for more insights.

In an episode of "Grafana's Big Tent," hosts Mat Ryer, Matt Toback, and Tom Wilkie explore the complex concept of observability, distinguishing it from traditional monitoring, which mainly focuses on metrics and time series data. Observability is described as the ability to understand the behavior of systems through various data types like metrics, logs, and traces, allowing deeper insights into system performance and issues. The discussion emphasizes the importance of monitoring on Service Level Objectives (SLOs) rather than just metrics, and highlights the challenges and value of distributed tracing, despite its high adoption cost. The episode also touches on the pitfalls of over-relying on dashboards and the necessity of strategic alerting to avoid overload, advocating for alerting on symptoms rather than causes, except in critical cases like disk space. The conversation demonstrates how observability can enhance system performance through methodologies like the RED method and error budgets, offering insights into effectively managing complex system architectures.