December 2019 Summaries

In 2019, Grafana Labs experienced significant growth and achievements, including celebrating its fifth anniversary and more than doubling active installations of Grafana to 482,000. The team expanded from 37 to 105 members across 22 countries, and the company raised $24 million in Series A funding to enhance its vision of an open, composable observability platform. Notable additions to the team included key roles such as Douglas Hanna as COO, Dave Kranowitz as VP of Global Sales, and Ryan McKinley as VP of Applications. The company also marked the general availability of Loki with version 1.0.0, engaged in 28 speaking events including keynotes and talks at KubeCon, and published 177 blog articles. Additionally, Grafana Labs introduced sustainable swag through carbon offsets at prominent industry events like PromCon, KubeCon NA, and AWS re:Invent, setting a positive tone for the coming year.

As 2019 concluded, Grafana Labs reflected on the year's most-read blog posts, highlighting topics ranging from technical insights to company milestones. These included a detailed account of a production outage in Grafana Labs' Hosted Prometheus service and preventative measures taken afterward, as well as a preview of new visualizations in Grafana v6.2. Other popular posts covered GitLab's public dashboards, a personal home dashboard setup, and public Grafana dashboards from organizations like Wikimedia and CERN. Additionally, guides on creating display kiosks for Grafana dashboards and the design behind Loki, a log aggregation system, were featured. The series also included a talk on visualization best practices and an announcement about Grafana Labs raising $24 million in Series A funding, which was intended to further their commitment to the open-source community.

Marcus Olsson, a Developer Advocate at Grafana Labs based in Stockholm, Sweden, plays a key role in helping users and contributors engage with Grafana through talks, workshops, blog posts, tutorials, and code samples. In addition to his role at Grafana, Marcus has contributed to open source with his notable project, tui-go, a UI library for terminal applications, although he eventually stepped away from its maintenance. Outside work, Marcus enjoys learning new things, with current interests in improving his cooking skills and exploring zero-waste practices, alongside hobbies like playing board games and video games. He has a penchant for Chillstep music and finds value in using the Headspace app for relaxation. Marcus humorously expresses a desire for a superhero power to absorb book contents by touch, acknowledging its potential villainous undertones.

Tom Wilkie's blog post discusses advancements in optimizing Prometheus query language (PromQL) for high-cardinality data and single-thread limitations. At KubeCon San Diego, Wilkie presented techniques to improve query performance by leveraging time-based parallelization, caching, and aggregation sharding, which significantly reduce query latency. These methods enable executing PromQL queries with lower latency, crucial for applications like capacity planning and performance analysis. The innovations are applied in Grafana Cloud's Prometheus service, based on the open-source Cortex project, which aims for full Prometheus compatibility. Despite Prometheus's single-threaded design, these enhancements demonstrate how parallelization and sharding can be effectively implemented, benefiting not only Cortex but also other systems like Thanos. The post provides a walkthrough to test these improvements, showing a significant reduction in query execution time and highlighting ongoing efforts to enhance query planning and execution across the Prometheus ecosystem.

Aengus Rooney, a Solutions Engineer at Grafana Labs, plays a key role in integrating Grafana's software with client environments and contributing to open-source projects like ASF, CNCF, Prometheus, and Loki. Based in the Greenwich Meridian time zone, Aengus enjoys reading and exercising in his free time, engaging in activities such as jogging, cycling, and swimming. He also has a penchant for electronic and progressive house music, which aids his concentration while coding, preferring spaces over tabs due to past challenges with code portability. A fan of IoT projects and gadgets, he is currently involved with home weather stations. In his leisure time, Aengus appreciates binge-watching shows like Mr. Robot and indulges in matcha and sesame ice cream.

Navigating dashboards efficiently is crucial for maintaining clarity and usability, given that complex dashboards often require users to switch between different views to get the complete picture. Grafana offers three types of links—Dashboard Links, Panel Links, and Data Links—to streamline this process by creating shortcuts to related dashboards or external websites. Dashboard Links allow users to jump to another dashboard with the same context, including time ranges and template variables, while Panel Links offer more granular navigation by linking specific panels to detailed dashboards. Data Links, a more recent addition, enable even finer granularity by incorporating series names or point values into links, particularly useful for Graph, Gauge, and Bar Gauge visualizations. By understanding how these different types of links can optimize navigation, users can enhance their workflow and ensure they are always viewing the right data in the right context.

Grafana and Loki enable automatic annotations on Grafana dashboards by using Loki queries to correlate events with metrics, facilitating better event tracking and analysis for operators and developers. Since version 6.4.0, Grafana has allowed the use of Loki queries to automatically generate annotations, where each log line from a query is displayed as an annotation at the correct time. By setting up an annotation query in Grafana's dashboard settings and selecting a Loki data source, users can specify log lines to appear as annotations, such as changes in a Kubernetes deployment. The kubernetes-diff-logger application logs changes to Kubernetes objects, and Promtail can be used to extract JSON fields as Loki labels, making them available for direct querying. This setup helps users correlate changes in their deployments with metrics such as memory usage, latency, or success rates, providing clear visual markers on Grafana dashboards. The technique is especially useful for tracking infrastructure changes in Kubernetes environments and is open to contributions from those interested in expanding the capabilities of the kubernetes-diff-logger project.

Ward Bekker, a Solutions Engineer at Grafana Labs, plays a pivotal role in assisting prospects and customers to maximize their use of Grafana products by addressing technical inquiries and collaborating with engineering teams for complex issues. Living in a village between Amsterdam and The Hague, Bekker contributes to open-source projects like Loki and has a history with Apache Metron and Travis-CI. In his free time, he enjoys spending time with his family in nature, running long distances, and utilizing his Topo Standing Desk Mat for comfort. Bekker prefers coding with a mix of music and values uninterrupted time for productivity, favoring strawberry cheesecake ice cream and having a unique ability to fall asleep at will, which he considers his superpower.

The storage engine of Prometheus 2.x, known as TSDB, has undergone significant advancements over the past year, highlighted at PromCon 2019 by Ganesh Vernekar. Initially developed independently, TSDB has now been integrated into the Prometheus repository and has seen contributions from over 60 developers. Key developments include the resolution of overlapping TSDB blocks that enabled backfilling, the use of Google's Snappy algorithm for WAL compression reducing WAL size by up to 50%, and various optimizations that have improved query execution time and reduced memory usage. Enhancements also include a new TSDB CLI command for analyzing data churn and cardinality, and the introduction of a read-only mode. Future plans involve implementing isolation in TSDB to achieve full ACID compliance, lifting index size limits to accommodate larger data sets, and improving WAL checkpointing to enhance performance and reduce memory usage. These innovations aim to make Prometheus more efficient and scalable, with ongoing community contributions playing a crucial role in its development.

In the blog post, Ivana Huckova shares her hands-on approach to learning Prometheus, an open-source monitoring system, by starting with simple "hello world" projects. She describes setting up Prometheus and using Node Exporter to monitor her computer system, Prometheus Middleware to track an express application's performance, and GitHub Exporter to observe GitHub repositories. Each project involves configuring Prometheus to scrape metrics, which are then visualized in Grafana, a popular analytics platform. Huckova emphasizes the importance of exporters, which are scripts or services that collect specific metrics, and provides guidance on integrating these tools to showcase Prometheus's capabilities. The blog serves as a practical guide for beginners looking to explore Prometheus through small, manageable projects.

KubeCon featured a discussion on cloud-native architecture, examining the challenges and benefits of monoliths and microservices. Grafana Labs engineers Goutham Veeramachaneni and Edward Welch introduced the concept of "monomicroliths," which combine elements of both architectures to address scalability and simplicity issues. They highlighted the ease of developing monolithic applications but noted the complications that arise as businesses expand, necessitating a shift to microservices for better scalability. However, the complexity of managing numerous microservices can become overwhelming, leading to inefficiencies. The monomicrolith approach offers a solution by consolidating microservices into a single binary application that can scale horizontally and be managed easily, allowing for incremental development and simplified troubleshooting. This approach has been successfully applied to projects like Cortex, Thanos, and Loki, enhancing adoption and development processes. The strategy involves using modular code bases and gRPC for communication, retaining a monorepo structure for consistency, and balancing trade-offs in performance and flexibility, ultimately making the development process more seamless and gradual.

Grafana Labs has shifted its focus from Cortex to Grafana Mimir, a scalable, open-source long-term storage for Prometheus, while continuing to share insights into optimizing Cortex for maximum performance. During a KubeCon session, engineer Goutham Veeramachaneni explained how to configure Cortex for scale, emphasizing its high configurability with over 1,000 lines of help documentation and numerous flags for fine-tuning components. Key strategies include using caching mechanisms, managing the write and read paths effectively, and setting appropriate limits and alerts to prevent overloads and ensure smooth operation. Veeramachaneni highlighted the importance of dashboards like the Cortex Writes and Reads Dashboards for monitoring system health and performance, offering a structured method to diagnose and address issues quickly. Grafana Labs also recommends setting strict limits on queries and memory usage to prevent disruptions, and they employ tools like Jaeger for distributed tracing to enhance system visibility. Despite challenges, Veeramachaneni noted that with proper configuration and alert management, operating Cortex has become more manageable, reducing unnecessary paging and improving overall system reliability.