July 2020 Summaries

At Grafana Labs, the integration of Scylla, an open-source Cassandra-compatible backend, with Jaeger tracing has demonstrated impressive performance capabilities, handling approximately 175,000 writes per second despite being deployed with default settings on Google's Kubernetes Engine. The disparity between the high number of writes and the lower span ingestion rate, around 8,000 spans per second, is attributed to the indexing process in Jaeger, which writes multiple rows per span to support detailed search capabilities. To manage backend load while maximizing span ingestion, Grafana Labs recommends adjusting sampling rates using Jaeger's remote sampler and utilizing configuration options to control indexing, thereby reducing unnecessary writes. Implementing a tag blacklist, disabling certain indexes, and using "firehose" mode for spans with less indexing needs have been effective strategies, resulting in a significant reduction in backend writes. Overall, leveraging these configurations has allowed Grafana Labs to maintain high performance and stability in their distributed tracing infrastructure.

Grafana Labs has transitioned from contributing to Cortex to launching Grafana Mimir, an open-source long-term storage solution for Prometheus that supports massive scalability. Cortex, historically used for distributed storage, offers horizontal scalability and high availability for Prometheus, but its architecture can lead to operational complexities. To address these challenges, a new storage engine for Cortex was developed, based on Prometheus TSDB and Thanos core components, eliminating the need for an index store while maintaining scalability and performance. This new blocks storage architecture, though still experimental, is being tested at scale and shows promising results in reducing complexity compared to the traditional chunks storage. The development involved significant collaboration within the community and between Cortex and Thanos projects, highlighting an ongoing effort to improve the efficiency and reliability of Prometheus storage solutions.

Grafana Labs employs the practice of 'dogfooding,' using their own products internally throughout the development process to improve their offerings, particularly in the realm of Grafana Cloud alerting. By integrating new products within their own workflow in a 1:1 manner, without shortcuts, they uncover unique use cases and gather iterative feedback that propels product development and enhances customer experience. This approach also involves a controlled rollout strategy, starting with internal teams and select customers before a full-scale launch, allowing for rolling updates and feedback-driven improvements. The process not only fosters adaptability among engineers but also ensures that the products are refined and robust by the time they reach the end users. The commitment to using their own tools helps Grafana Labs maintain an engaging development process and promises a more effective alerting experience for their customers.

Recent advancements in the Prometheus storage engine, TSDB, have significantly optimized performance by reducing memory consumption and restart times, leading to a more efficient data management process. These improvements include memory-mapping of head chunks and reduction in memory usage by postings offset tables, which have enabled faster snapshotting of in-memory data and an 80% reduction in restart turnaround time. Additionally, ongoing efforts to develop a tool for bulk-importing data are underway, with support planned for OpenMetrics and CSV formats. Future enhancements include better histogram support, experimenting with various chunk encodings for improved disk space and query optimization, and exploring methods for maintaining constant memory usage during data compaction. These developments indicate a robust trajectory for TSDB, with more features and enhancements expected in the near future.

Ivana Huckova, a frontend engineer at Grafana Labs, works on the Observability squad focusing on the Explore feature, which aids in infrastructure monitoring and incident response by allowing users to delve into metrics, logs, and traces. Based in Bratislava, Slovakia, Ivana contributes to open-source projects like Grafana and Prometheus documentation, and she is active on GitHub and Twitter under the handle @ivanahuckova. In her free time, she enjoys hiking, skiing in Slovakia's mountains, traveling, reading, running with her newly adopted dog Kale, and engaging in tech projects like home and sourdough monitoring using Home Assistant and Arduino. Ivana is also a fan of podcasts such as 99% Invisible and Masters of Scale, and she recently binge-watched the show Sense8.

Prometheus has seen significant advancements since PromCon 2019, with updates including a new React-based UI in version 2.14, metadata API enhancements in 2.15, and the introduction of ACID database isolation in 2.17. Subsequent versions focused on improvements such as memory footprint reduction by 50% in 2.19 and native support for service discovery in 2.20. The project continues to evolve with a focus on metadata utilization, workflow changes, and backfilling support strategies. Prometheus has also adjusted its development approach due to COVID-19, opting for shorter, more frequent virtual dev summits to manage its growing backlog. Efforts are underway to improve documentation, with plans to restructure it into distinct user manuals, reference sections, and guides. The node_exporter has achieved version 1.0 with experimental TLS support, and community engagement remains a priority, reflected in discussions around mailing list accessibility and the potential for a tech writer to enhance documentation.

The updated New Relic plugin for Grafana introduces several enhancements, including the integration of New Relic as a data source for Grafana Cloud and Enterprise users, allowing them to visualize and transform data without relocating it. The plugin now supports the use of the New Relic Query Language (NRQL) for creating template variables, enhancing query capabilities with regex filtering and requiring a Personal API key for configuration. Additionally, the plugin adds alerting functionality, allowing users to set up alert rules for graph panels based on calculated thresholds, improving data-driven decision-making processes. These updates, along with the Grafana 7.0 transformations, enable the creation of actionable dashboards that correlate New Relic data with other metrics, logs, and traces, further enhancing the observability capabilities of Grafana users. For those interested, more details and support are available through the New Relic solutions page and by contacting their team.

The tutorial provides a comprehensive guide on setting up Promtail on Amazon EKS to achieve centralized log visibility using Loki and Grafana. It covers the installation and configuration of necessary tools like eksctl, AWS CLI, kubectl, and a Grafana instance with a Loki data source. The process entails creating a Kubernetes cluster on EKS, deploying Promtail as a DaemonSet for log collection, and ensuring service discovery compatibility with Prometheus. Users are guided through configuring log forwarding to Loki, including systemd logs and Kubernetes events, and utilizing Helm for deployment tasks. The tutorial emphasizes real-time log exploration in Grafana, highlighting the seamless integration of metrics and logs for efficient troubleshooting and monitoring within Kubernetes environments.

Grafana Labs is hosting two informative webinars featuring technical experts to explore the capabilities of its tools, focusing specifically on Loki logging configuration and Grafana plugins. The first webinar, "Logging with Loki: Essential Configuration Settings," is scheduled for July 21, 2020, and will be led by Loki project maintainers who will delve into setting up Promtail and Docker drivers, configuring the Loki server, and managing storage backends, with the prerequisite of having watched an introductory session. The second webinar, "All About Grafana Plugins: Visualizing Disparate Data Sources in One Place," will occur on July 22, 2020, and will be conducted by Grafana Labs Solutions Engineers, who will demonstrate how to leverage data source plugins to integrate various platforms like Elasticsearch and ServiceNow into Grafana, highlighting the unique features of Enterprise plugins and strategies to optimize data visualization and dashboard creation.

Grafana v7.1 introduces several enhancements and new features, building on the developments from Grafana 7.0. Key updates include support for the Flux query language in InfluxDB data sources, feature parity for the new table panel with the introduction of merge on time transformation, and the removal of the query mode selector in Explore for a more unified querying experience. The update also introduces query history search and internal linking for Elasticsearch, as well as the ability to provision app plugins. Enhancements in data sources such as Azure Monitor and Google Cloud Monitoring include support for multiple dimensions and deep linking to the Metrics Explorer in Google Cloud Console. The release also offers new functionalities in the time range picker and stat panel and extends Grafana Enterprise features with support for HashiCorp Vault and monthly report scheduling. Users are encouraged to upgrade to v7.1 to take advantage of these improvements.

Cortex and Thanos, two open-source projects designed to scale Prometheus, have evolved to collaborate closely, leveraging each other's strengths to enhance their capabilities. Initially, Cortex aimed for high scalability and performance, while Thanos prioritized operational simplicity and cost-effectiveness. Over time, they learned from each other, with Thanos adopting a push-based model and Cortex simplifying its operations. Their collaboration has involved shared improvements, such as the integration of caching techniques and configuration optimizations, and the development of the Cortex blocks storage, which utilizes Thanos components. This partnership has not only enhanced both projects but also contributed to the broader open-source community, demonstrating the value of collaboration in overcoming technical challenges and fostering innovation.

Gardener, an open-source Kubernetes-as-a-service project by SAP, is transitioning its logging stack to the Loki system to overcome inefficiencies and limitations posed by its previous architecture, which used Fluentd and Elasticsearch. Gardener, which manages thousands of Kubernetes clusters worldwide for SAP and other community users, faced challenges with high resource consumption and complex tracing issues with the former setup. The new logging architecture leverages Fluent Bit and Loki to enhance resource efficiency, scalability, and user experience by integrating logging and monitoring into a unified Grafana interface. This change was inspired by insights gained from KubeCon North America and is currently being tested and implemented, with promising preliminary results, in anticipation of deployment across production environments.

PromCon Online, a conference dedicated to the Prometheus monitoring system, features sessions highlighting significant developments and collaborations in the open-source community. Grafana Labs, which recently shifted its focus from Cortex to Grafana Mimir, a scalable long-term storage solution for Prometheus, has team members presenting on various topics. These include a joint session with Red Hat discussing the collaborative evolution of Cortex and Thanos projects, a review of recent enhancements to the Prometheus storage engine by Ganesh Vernekar, and insights into the future of Prometheus and its ecosystem by Richard Hartmann. The event underscores the collaborative spirit and continuous evolution of open-source projects within the cloud-native landscape.

Setting up Promtail on an AWS EC2 instance can greatly enhance log management by aggregating and sending logs to a Loki instance for analysis. This tutorial provides a detailed guide on configuring Promtail, starting with the prerequisites such as having an AWS account, a VPC, and a Grafana instance with a Loki data source. The process involves creating an EC2 instance, setting up a security group, and using AWS CLI commands to manage resources. Promtail is installed and configured to scrape logs using Prometheus scrape configurations, which are particularly useful for EC2 service discovery, making it easier to manage and query logs with attached metadata. The guide also explains how to configure Promtail as a systemd service, ensuring it runs automatically and reliably, and demonstrates how to use Grafana to verify that Loki receives the logs. Additionally, it covers the integration of systemd logs into Promtail's configuration to scrape logs from journald, ensuring comprehensive log management within a cloud environment.

Annanay Agarwal, a developer at Grafana Labs based in Bangalore, contributes to the Cortex project and manages internal tracing infrastructure. His journey in open source began with a Google Summer of Code project improving TensorFlow kernel performance, leading to contributions in the observability space with JaegerTracing and OpenTelemetry, and now focusing on Cortex. Annanay's interests extend beyond work, as he enjoys reading, music, Netflix, and sports like football and badminton. During the Coronavirus lockdown, he picked up cooking and aspires to contribute to the Loki project due to his admiration for it. His GitHub handle is annanay25 and his Twitter handle is @mrannanay. Annanay prefers coding in silence, occasionally accompanied by pop music, and if he could have a superpower, it would be the ability to fly, reflecting his fascination with airplanes.

In Joe Elliott's guide on diagnosing dropped spans in Jaeger distributed tracing, the focus is on understanding and addressing the loss of critical spans within the Jaeger span ingestion pipeline. The guide highlights the importance of refreshing the Jaeger UI before diving into infrastructure diagnostics, as traces might not be fully ingested initially. It details the Jaeger setup from the in-process client to the Jaeger Agent, Collector, and Ingester, emphasizing the use of metrics such as jaeger_tracer_reporter_spans_total and jaeger_agent_client_stats_spans_dropped_total to identify where spans are being lost. The guide suggests increasing queue sizes, reducing latency, and optimizing the placement of agents and collectors to mitigate span loss. It also discusses the potential issues with using UDP for communication and recommends reviewing network metrics if packet loss is suspected. The guide concludes by underscoring the role of effective queue management in reducing dropped spans and encourages exploring additional metrics provided by Jaeger for a more comprehensive understanding of the system's behavior.

Grafana and NGINX have partnered to enhance visibility and observability for the open-source community by tightly integrating their tools. This collaboration aims to provide a seamless experience for users by combining Grafana’s visualization capabilities with NGINX’s metrics and logging features, resulting in a comprehensive monitoring solution. Grafana has become a leading tool for visualization in the open-source market, while NGINX, now under F5's ownership, focuses on providing extensive metrics and logging insights through tools like Prometheus and Loki. The partnership intends to reduce the time to value and friction for users by offering an integrated platform that brings together metrics, logs, and tracing, thus enabling users to efficiently monitor and manage their application environments. By leveraging NGINX's Prometheus exporter and Grafana's visualization capabilities, users can gain a complete picture of their systems' performance and health. The joint effort also emphasizes the importance of observability in managing distributed systems, highlighting the need for a cohesive approach to handle the increasing volume of data signals coming from various sources.

Grafana 7.0 introduces new Enterprise features designed to enhance user experience and manage large installations through usage insights and a user presence indicator. Usage insights provide detailed data on dashboard usage, errors, and user interactions, helping organizations identify popular or problematic dashboards while minimizing data storage costs through deduplication and growth limits. The user presence indicator, inspired by collaborative tools like Google Docs, allows users to see who else is engaging with a dashboard, fostering communication and collaboration, especially during incidents. Grafana's updates aim to ensure these features are seamlessly integrated into the overall experience, with improvements to search functionality and dashboard interfaces. Looking ahead, Grafana plans to prioritize data source insights and refine data collection methods while maintaining performance across different resource environments.

The Grafana Cloud Agent, introduced as a remote_write-focused Prometheus agent, is a streamlined version of Prometheus that excludes querying and local storage but retains service discovery, relabeling, WAL, and remote_write capabilities. With its v0.4.0 release, it supports "integrations," which are embedded exporters that facilitate metric scraping and remote writing without manual configuration, simplifying setup. The two initial integrations are node_exporter for UNIX-based hosts and agent for self-scraping, with the possibility of manual configuration for additional exporters like blackbox_exporter. This setup provides a Prometheus-like environment for metric retrieval and transmission to endpoints such as Grafana Cloud, streamlining configurations and enhancing ease of use. Future releases aim to expand integration options, further reducing the complexity of obtaining and managing metrics within the Grafana ecosystem.