August 2019 Summaries

Kyle Brandt is a backend engineer at Grafana Labs, based in Florida, USA, working primarily on the development of Grafana's backend. He has contributed to open-source projects like scollector and Bosun in the past and is currently focused on Grafana. His GitHub and Twitter handles are kylebrandt and @kylembrandt, respectively. Outside of work, Kyle is learning to play the piano and enjoys listening to sci-fi audiobooks. He recently binge-watched "Good Omens," which he also enjoyed as an audiobook. He values code consistency and uses the go fmt tool for formatting.

Grafana has released versions 5.4.5 and 6.3.4 to address a significant security vulnerability, identified as CVE-2019-15043, which affects all versions from 2.0.0 to 6.3.3. This vulnerability allows unauthorized access to parts of the Grafana HTTP API, enabling potential denial of service attacks. The issue was reported on August 12, 2019, and following a thorough process of confirmation and patch development, the updated versions were released on August 29, 2019. Grafana Cloud instances have already been patched, and Grafana Enterprise customers have received updated binaries. Users running affected versions are advised to upgrade immediately to ensure security. Further, Grafana maintains a Security Announcements category on its community site for updates and encourages reporting any vulnerabilities to [email protected].

Grafana 6.3 introduces enhanced features for data interaction, particularly focusing on improving cross-system navigation and usability through the introduction of Panel links and Data links. These new functionalities aim to simplify the integration between Grafana and external systems by enabling users to recreate the context of their data visualized in Grafana in external systems. Data links, specifically in the Graph panel, offer access to template variables, the current time range, and series-specific variables such as series name and clicked datapoint timestamp, thus allowing for more precise contextual recreation. This update stems from collaboration with Bloomberg and feedback from the Grafana community, addressing previous limitations in linking metrics visualized in Grafana with logs in other systems like Kibana. Additionally, Data links are now more accessible via a context menu in the Graph panel, and future updates plan to extend these capabilities to other visualizations and include more series-specific variables. Grafana Labs continues to explore new ways to enhance cross-system interaction and encourages users to stay updated through their blog and social media channels.

Grafana Labs undertook a performance optimization project for their product by first conducting an audit using Google's Lighthouse suite to identify key areas needing improvement, particularly focusing on the size of JavaScript resources being downloaded. The audit revealed that a significant portion of resources was unused, prompting the use of webpack's dynamic imports to implement lazy loading for infrequently used large packages and route-based code splitting. These changes led to noticeable improvements in performance metrics such as First Contentful Paint and Speed Index. The optimization process also included adopting HTTP/2 and making additional refinements based on a GitHub repository's recommendations for optimizing common dependencies. Future plans include reducing HTML size and cleaning up CSS to further enhance performance.

Grafana Labs introduces Ed Welch, a software engineer working on the Loki project, as part of their team profile series. Based in Rochester, N.Y., Ed is deeply involved in open-source projects, primarily focusing on Loki, and also maintains a personal project related to his home weather station on GitLab. Known online as "slim-bean," a nickname from his grandfather, he tends to avoid social media but does have a Twitter account created to address a grievance with a company. In his leisure time, Ed enjoys playing volleyball, backpacking, walking his dog, and experimenting with home automation. He is enthusiastic about technology, recently acquiring a Raspberry Pi 4 and integrating a Raspberry Pi Zero with his electric Nissan Leaf to monitor and control its charging. Ed prefers using spaces over tabs in coding due to the default settings in most editors.

Ed Welch's guide explores the integration of monitoring tools OSSEC, Loki, Prometheus, and Grafana on a Raspberry Pi to enhance home network security and visualize alerts. The setup begins with OSSEC for real-time email alerts and evolves to using Loki, Prometheus, and Grafana for a comprehensive view of these alerts, eliminating the need for manual log searches. Welch details the installation and configuration processes for each component, emphasizing that the project is experimental and not production-ready. The tutorial includes creating systemd services to ensure components run on boot, with a focus on simplifying and visualizing network activity through Grafana dashboards. The guide concludes with steps to import pre-built dashboards into Grafana, ultimately allowing users to run a scalable monitoring system on a cost-effective Raspberry Pi.

Grafana Labs effectively integrates Loki with Kubernetes events to enhance debugging and logging capabilities, particularly during outages caused by configuration issues or node restarts. Kubernetes events provide valuable insights into cluster changes but are only retained for an hour, prompting Grafana Labs to explore using Loki for persistent storage and querying of these events. They utilize the eventrouter, a tool initially developed by Heptio, to extract Kubernetes events and send them to Loki, allowing for better management of logs by adding namespace labels via Promtail's pipeline configuration. This integration with Loki, a Prometheus-inspired log aggregation system, facilitates the efficient storage, search, and correlation of logs and metrics. Despite the successful implementation, Grafana Labs acknowledges the need for improving the user interface when dealing with JSON logs, as the current format can be overwhelming when searching for specific data points. Over time, Loki has gained popularity, approaching general availability while Grafana Labs continues to enhance its features, including improved log detection, Docker plugin support, and structure for unstructured logs.

Loki, a Prometheus-inspired logging service launched by Grafana Labs, optimizes storage, search, and aggregation, allowing users to explore logs seamlessly in Grafana. Designed to operate as both microservices and monoliths, Loki efficiently correlates logs and metrics to reduce costs. As it approaches general availability, Grafana Labs is enhancing Loki's query optimization capabilities, focusing on ingestion retention and label queries. The service uses ingesters to compress writes, significantly reducing backend load, and has transitioned to using backend databases for label queries to manage memory usage effectively. Although compression speed has improved with the switch from Go's gzip to the compress package, decompression remains CPU-intensive, prompting future plans for query parallelization and exploring alternative compression algorithms. All developments are open-source, and Grafana Labs continues to refine Loki's performance with contributions from its community, aiming to enhance its efficiency in handling large-scale log data.

Loki, a Prometheus-inspired service launched at KubeCon North America, optimizes storage, search, and aggregation of logs, seamlessly integrating with Grafana for easy log exploration. As Loki nears general availability, significant efforts have focused on query optimization, specifically addressing memory consumption during log processing. The service employs the Iterator pattern to decouple algorithms and manage log data efficiently, introducing innovations like lazy iterators and chunk batch iterators to handle high cardinality queries without overwhelming system resources. Additionally, recursion in iterators was eliminated to prevent excessive memory use due to Go's stack size constraints. These enhancements aim to streamline log management, ensuring low memory usage across queries, with future updates to address ingestion retention and label queries.

Cyril Tovena is a software engineer at Grafana Labs, based in Montréal, Canada, working primarily on open-source, distributed systems like Loki and Cortex, with contributions focused on performance optimization and integrations. He is also involved with other projects, such as Agones, which scales game servers on Kubernetes. In his free time, Cyril enjoys playing video games and soccer with his son, hiking, and skating with his dog, and he prefers to code while listening to folk or indie music. A fan of technology, he appreciates his 4k Retina iMac and recently binge-watched Stranger Things 3. Cyril humorously cites being a dad as his superhero power and admits to favoring tabs in coding, although tools like gofmt make formatting simpler.

Loki, a Prometheus-inspired logging service launched by Grafana Labs, is approaching general availability as it optimizes the storage, search, and aggregation of logs, making them easily accessible in Grafana. The service supports both microservices and monolithic applications and aims to correlate logs and metrics efficiently. The first installment of a multi-part series focuses on query optimization efforts to enhance Loki's CPU and memory usage by improving how log data is processed. Loki stores log data in compressed "chunks," which can contain over 100,000 lines, leading to challenges in resource consumption when handling numerous chunks. The optimization process involves techniques such as using the pprof tool and go benchmarks to address memory and CPU usage issues, including dealing with Go's garbage collector and optimizing memory allocation through object pooling. The series also discusses reducing unnecessary allocations during regex filtering and strategies for identifying and resolving memory leaks. Future installments will delve into iterators, ingestion retention, and further enhancements to Loki's capabilities.

Grafana v6.3 introduces a new log row context viewer for Loki's log exploration workflow in Grafana Explore, enhancing the debugging experience by allowing users to view surrounding log lines for better context. Loki, launched at KubeCon North America and inspired by Prometheus, is designed to optimize log storage, search, and aggregation while being easy to use with both microservices and monoliths. Its integration with Grafana Explore enables users to correlate metrics and logs, providing a more seamless debugging workflow by using filter expressions to narrow down log results. The new "Show Context" feature allows users to easily view the log lines surrounding a specific log line, which can be crucial for understanding errors, especially when on call. The context feature is available only for queries with search terms and is based on the same label set as the original query, requiring adjustments in query selectors if logs from more hosts are needed. This enhancement aims to improve the debugging workflow, and user feedback is encouraged.

Loki, a Prometheus-inspired logging service launched at KubeCon North America, is nearing general availability, with its live tailing feature being a major highlight. Designed to work with both microservices and monoliths, Loki efficiently correlates logs and metrics, saving users money by optimizing storage, search, and aggregation. Live tailing, a highly requested feature, allows users to view logs in real-time as they enter the system, aiding in the quick identification of bugs and providing a live state of services. It employs gRPC streams for efficient communication, ensuring resilience against failing ingesters while supporting filters and expressions for tailored log viewing. The Log CLI tool enhances functionality by allowing logs to be queried and manipulated in various formats. For real-time log viewing, users can select "Live" from the refresh dropdown in Grafana, with proper websocket configurations. Additional features such as loki-canary for early detection of missing logs and Docker logging driver plugin support further enhance Loki's capabilities.

Loki's version 0.2.0 release marks a significant milestone in its development, driven by substantial community contributions and improvements in performance, features, and the release process. The delay in the release was largely due to prioritization issues, but with the involvement of new contributors like sh0rez, the release process has been refined, setting the stage for more frequent future updates. Notable enhancements include improved query performance and memory usage, a more robust log processing pipeline, and new features like a Docker plugin and the loki-canary application for monitoring log accuracy. Efforts have also been made to improve build and release tooling, allowing for the generation of multi-architecture container images and binaries. The project aims for bi-weekly to weekly releases as it progresses toward a general availability (GA) release, with additional plans to enhance community engagement and governance.

Tom Braack, an intern at Grafana Labs, is currently contributing to the development of open-source projects like Grafana Loki and tanka during his five-week summer holidays after being invited to join the company by Goutham at KubeCon 2019. Residing in a small German town, Tom is passionate about enhancing user and developer experiences through bug fixes and feature improvements, and he took the initiative to create a minimal replacement for the deprecated ksonnet, which was a configuration utility for Kubernetes. In his free time, he enjoys engineering projects using Raspberry Pis and ESP8266s, exploring opportunities for building smart home devices, and is an advocate for using spaces over tabs in coding. Tom is also active on Twitter and GitHub under the handle @sh0rez, enjoys binge-watching popular series like Money Heist, and has varied preferences for ice cream flavors, depending on the day.

Cortex, an open-source, horizontally-scalable, and highly-available clustered implementation of Prometheus, celebrated its first official release, led by Chris Marchbanks from Splunk, despite not introducing any new features, as users desired stamped versions. Initially developed over three years ago and already in production at companies like EA, Aspen Mesh, Mayadata, and Gojek, Cortex had previously followed a continuous deployment approach rather than a formal release cycle. The project, which is instrumental for Grafana Cloud's Hosted Prometheus and has helped accelerate the development of Grafana’s log aggregation system Loki, was accepted into the CNCF Sandbox and has seen contributions from over 50 individuals. As it continues to grow, Grafana Labs is enhancing Cortex’s scalability, integrating Prometheus’s TSDB for on-premises use, and improving cluster scalability processes, alongside plans to bring PromQL acceleration techniques to Prometheus and Thanos, making it an exciting time for those involved in developing distributed timeseries databases.

The article provides insights into frequently asked questions from the Grafana Labs community, highlighting solutions for using Grafana, MySQL, and Elasticsearch. It features queries and techniques shared by Daniel Lee, Grafana Labs' Director of Software Engineering, to assist users in creating graphs from MySQL data, setting alternative metric names in graph panels for various data sources, and filtering Elasticsearch data by null values. The discussions illustrate how Grafana's features, such as the MySQL query builder introduced in version 5.4, enhance user experience by simplifying query creation. It also emphasizes Grafana Labs' collaborative approach, inviting users to participate in discussions on their community board.

Grafana v6.3 introduces several enhancements and new features aimed at improving user experience and functionality. Key updates include enhancements in the Explore feature, such as Loki Live Streaming and Context Queries, which allow for better log management. Additionally, the release adds support for Elasticsearch and InfluxDB logs in the Explore mode. Data links have been introduced to allow more dynamic data interaction, and a redesigned time picker simplifies accessing time ranges. The Graph display now supports gradients for improved visualization, and significant improvements to Grafana Enterprise include LDAP Active Sync and built-in SAML authentication, providing better user management and security options. These enhancements reflect Grafana's ongoing commitment to improving its platform, catering to both open-source and enterprise users.

Björn "Beorn" Rabenstein is a Principal Software Engineer at Grafana Labs, based in Berlin, Germany, and is primarily focused on working with Prometheus, both in external contributions to the project and as a consultant for its internal usage within the company. Known for his significant open-source contributions primarily to Prometheus, Björn ranks highly in both the number of contributions and his long-standing involvement in the project. Outside of work, he enjoys numerous hobbies, with board games being his favorite, and prefers a quiet environment for coding, although he is adept at ignoring ambient noise. His GitHub handle is beorn7, and he humorously questions the relevance of Twitter. Additionally, he leaves code formatting decisions like the choice between spaces or tabs to automated tools such as "go fmt."