September 2020 Summaries

Amazon Timestream, recently launched by AWS, is a fast, scalable, serverless time series database designed for IoT applications, and it can be seamlessly integrated into Grafana's observability dashboards. Grafana, known for its popularity in visualization technology, allows users to visualize native Timestream queries using SQL-style syntax directly. Ryan McKinley, VP of Applications at Grafana Labs, highlights the company's dedication to collaborating with technology leaders to develop plugins that align with users' objectives, expressing excitement about the potential of Amazon Timestream as a data source for IoT and time series data. The collaboration aims to enhance user experience by leveraging Timestream's capabilities within Grafana, inviting users to explore and provide feedback on this integration.

Grafana 7.2 introduces the $__rate_interval variable, enhancing the accuracy and ease of configuring rate queries in Prometheus by automatically ensuring the range used is at least four times the scrape interval. This development addresses common challenges in Prometheus query language (PromQL) related to setting appropriate query ranges, which often resulted in no data being displayed due to insufficient range selection. The new variable helps overcome issues such as the need to manually adjust the Min step in each panel and limits on resolution requests from Prometheus, ultimately providing a more reliable and user-friendly experience for visualizing metrics. By extending the usual $__interval with an additional scrape interval, $__rate_interval ensures complete coverage of all counter increases, even when facing potential alignment issues within PromQL range selectors. This improvement encourages users to maintain consistent scrape intervals across their organization, simplifying the process and enhancing the reliability of their data visualizations.

Integrating performance tests into CI pipelines using GitLab and Grafana k6 allows for early detection of performance issues, leading to more stable and efficient applications. This tutorial guides users through incorporating load testing with k6 into GitLab, detailing how to create and run performance scripts that simulate virtual user activity and evaluate service-level objectives (SLOs). The setup involves configuring GitLab CI to execute both local and cloud-based tests, leveraging Docker images and environment variables for seamless testing from multiple geographic locations. Users are taught to set pass/fail criteria using k6 thresholds, which inform the CI tool of test outcomes, ensuring system reliability. Additionally, the tutorial explains how to schedule tests to run during off-peak hours using GitLab's pipeline scheduling feature, thus optimizing test execution without disrupting regular operations. Through these processes, users can achieve comprehensive load testing that accommodates various scenarios and environments, enhancing the overall performance testing strategy.

Richard Lam, a product manager for Grafana Cloud based in Washington, D.C., is passionate about creating a product that users love by understanding their needs and challenges. His role involves developing product vision and strategy, collaborating with various teams within Grafana Labs, and engaging with customers to gather valuable feedback. Richard enjoys translating complex data into actionable information and has a background in building observability tools, which made Grafana Labs a perfect fit for him. Outside of work, he has a keen interest in food, particularly cooking and exploring new ingredients, and has recently delved into the art of coffee brewing, favoring cold coffee and experimenting with different beans and brewing methods. Despite his culinary interests, Richard's favorite food remains a simple yet perfectly cooked steak, highlighting his appreciation for the basics of food.

Jonathan Stines, a Penetration Tester at Rapid7, has been using Grafana and Prometheus to monitor his 3D printing projects while quarantining at home. Inspired by the dashboards featured in the HBO series Silicon Valley, he set up his own system using OctoPrint plugins to export data via Postgres, InfluxDB, and Prometheus, all of which Grafana supports. This setup not only allowed him to create visually appealing dashboards to impress his friends but also to track the cost and filament usage of his prints. He enhanced his monitoring setup by incorporating a Raspberry Pi touchscreen for quick job status updates and historical print data. Stines credits online resources and guides for helping him achieve this setup and plans to expand his home automation efforts using Grafana, including integration with Home Assistant and VMware ESXi. Among his 3D creations are a skull desk organizer, mini Groot figures, a miniature poodle, a GoPro handle for his father, and a pink PETG project for his mother.

Grafana v7.2, released on September 23, 2020, introduces several new features and enhancements, building on previous versions such as Grafana 7.0. Key updates include customizable date formatting options, enabling users to adjust how dates appear across various visualizations by modifying server-wide settings. The release also brings new field options, table column filters, and advanced field override selection options, which have moved out of beta after extensive testing and feedback. Users can take advantage of new transformations, such as a Group By transformation and the ability to drag transformations and queries to reorder them. The update includes improved query inspection in Explore and the addition of a new variable for Prometheus, $__rate_interval. Additionally, sensitive alert channel settings are now encrypted for enhanced security. Grafana Enterprise sees the introduction of new report features, including a grid layout option and the ability to generate reports with different time ranges than what is shown on dashboards. The release also enhances the Azure Data Explorer plugin with a visual query builder to simplify KQL queries. Community contributions were acknowledged, and users are encouraged to explore these updates by downloading Grafana 7.2 or trying it in the cloud.

Grafana Labs has introduced several updates and products aimed at enhancing observability for large enterprises, including Grafana Metrics Enterprise (GME), which offers Prometheus-as-a-Service built on Cortex to enable scalable monitoring solutions. Additionally, a new Prometheus-style alerting feature for Grafana Cloud simplifies the alerting process without requiring extra tools or command-line interaction. The launch of the GitHub data source plugin allows users to visualize GitHub data, while updates to the Dynatrace plugin facilitate the creation of user-friendly dashboards from Metrics API v2 data. The article also mentions Grafana's recent community engagements, such as co-founder Torkel Ödegaard's appearance on a Kubernetes Podcast and the upcoming Grafana ObservabilityCON, which promises live talks and demos on open-source observability technologies like Prometheus and Loki. Furthermore, a Cloud Native Computing Foundation survey highlights Prometheus and Grafana as the top recommended observability technologies.

The Grafana GitHub plugin enables users to visualize and track data from GitHub’s API, allowing organizations to monitor outstanding pull requests effectively. This functionality is particularly useful for open-source projects like Grafana, Cortex, and Loki, where pull requests can easily be overlooked. Users can employ transformations and visualizations such as SingleStat to assess metrics like the average open time for pull requests and the total number of active pull requests. The plugin also supports the use of variables, which allows for the creation of more personalized and relevant dashboards, as well as annotations to correlate GitHub events with other metrics. However, users need to be mindful of GitHub's API limitations, including request limits and pagination, which can affect data retrieval. Future enhancements aim to provide better aggregation and historical data capabilities while addressing challenges related to API token limitations.

Éamon Ryan is a Solutions Engineer at Grafana Labs, residing in the Bay Area and working in the Pacific time zone. He engages with Enterprise customers to address technical challenges and improve their observability stacks, using Grafana's offerings and community resources. Ryan, a proponent of open-source technology, was drawn to Grafana Labs for its startup size and the opportunity to work with technology he appreciates. His contributions to open-source projects are primarily scripty, and he enjoys tinkering with his home lab in his free time. A fan of gaming and Korean BBQ, Ryan also enjoys using his Nanoleaf lights during Zoom calls and values the satisfaction of helping clients realize the potential of Grafana products. Known for his dad-joke humor, he humorously cites his ability to get sunburned quickly as a hidden talent, and he dreams of having control over time as a superpower.

Grafana Metrics Enterprise, now known as Grafana Enterprise Metrics, is a Prometheus-compatible service designed to address the complexities of using Prometheus at an enterprise scale, offering a simplified installation and configuration process for large organizations. Developed on the foundation of the CNCF incubated project Cortex, it provides a centralized, scalable solution for managing Prometheus metrics, including features like instance management, fine-grained access control, and authentication, enabling organizations to securely and efficiently handle metrics across multiple clusters. The service aims to overcome the challenges associated with Prometheus's single-process and pull-based architecture in enterprise environments, such as scalability, security, and network compatibility, while allowing for horizontal scalability and minimal system dependencies. Grafana Metrics Enterprise maintains full compatibility with the open-source Cortex project, adding enterprise-specific enhancements without detracting from the original open-source capabilities, and includes support and consulting services from Grafana Labs, which continues to contribute to the open-source community.

OAuth is a widely used authentication standard for verifying user identities via third-party services, and it's important for testing both anonymous and authenticated endpoints. The article explores how to load test APIs secured by OAuth using tools like Grafana k6, focusing on authentication methods utilizing Microsoft Azure Active Directory (AAD) and Okta. It delves into OAuth 2.0 authentication flows, highlighting the authorization code flow as the preferred approach and explaining the implicit flow for simplicity and testing. The text also discusses OAuth 2.0 grant types, specifically the client credentials grant for server-to-server authentication and the resource owner password credentials grant for user authentication. The article provides a practical example involving Microsoft AAD, including testing with curl and incorporating OAuth authentication into k6 load test scripts, while offering guidance on implementing and automating these processes.

Efforts are underway to enhance the use of the Loki logging system with AWS Lambda and other ephemeral services, which are challenging due to their short-lived nature and the difficulty in co-locating log forwarding agents like promtail with application logs. Promtail, now equipped with a Push API, can serve as an intermediary in log processing pipelines, facilitating the collection and ingestion of logs from AWS Lambda by assigning timestamps during ingestion rather than relying on those from CloudWatch, thus avoiding out-of-order errors. This method shifts cardinality issues from the number of log streams to the product of log groups and promtails, requiring unique labels for each promtail to prevent conflicts. While this approach mitigates complexities associated with ephemeral services, future plans include potentially relaxing Loki's ordering constraints to simplify log ingestion by eliminating the need to maintain strict chronological order, thereby reducing operational complexity for users.

Richard Lam, the new product manager for Grafana Cloud, introduces an improved feature for Prometheus-style alerting within Grafana Cloud, eliminating the need for external tools like cortextool or local Prometheus setups. The new centralized alerting feature allows users to manage, create, update, and delete alerts directly from a unified interface within Grafana Cloud, simplifying the process without requiring configuration file modifications. This update addresses user demands for centralized alert management, making it easier for teams to handle alerts without filesystem modifications on the Prometheus server. While this release enhances the user experience by making alert management more accessible and intuitive, it is just the initial iteration, with further improvements, such as integrating alert silencing, anticipated in future updates. Users are encouraged to provide feedback to continue refining and enhancing the functionality and overall product experience.

Grafana ObservabilityCON, scheduled for October 26-29, is preparing to host a series of tech talks and product announcements focused on observability, inviting community members to contribute by sharing their experiences and strategies. The call for proposals is open until September 22, encouraging participation from first-time speakers and underrepresented groups. The event aims to explore innovative uses of popular open-source observability tools like Prometheus, Loki, and Graphite, while also welcoming "non-tech" talks related to observability. Suggested topics include adoption strategies, scaling challenges, alerting, distributed tracing, migration stories, the use of Loki in production, and career paths in observability. The conference is an opportunity for practitioners to discuss their experiences, propose solutions, and offer advice on navigating observability challenges.

Grafana is organizing its first-ever ObservabilityCON from October 26-29, inviting the community to participate in tech talks and product announcements focused on observability. The event offers speaking opportunities for practitioners to share strategies, projects, and favorite open-source tools, with a call for proposals open until September 22. Encouraging submissions from first-time speakers and underrepresented groups, the conference aims to cover various topics such as adoption strategies, scaling challenges, alerting, distributed tracing, and career paths in observability. Attendees will have the chance to explore innovative uses of tools like Prometheus, Loki, and Graphite, and engage in discussions about both technical and non-technical aspects of observability. The event seeks to provide insights and inspiration for participants at all stages of their observability journey.

The tutorial explores the integration of k6, an open-source load testing tool, into Jenkins pipelines to enhance performance testing in CI/CD workflows. By incorporating k6 tests, developers can identify performance issues earlier, leading to more stable applications. The guide details the process of writing test scripts, setting thresholds for service-level objectives, and configuring Jenkins pipelines to trigger automated tests. It emphasizes the importance of defining performance thresholds to ensure application reliability and explains how to run tests locally or via Grafana Cloud for broader geographic coverage. Additionally, the tutorial covers setting up GitHub webhooks, Jenkins pipeline configuration, and using cloud tests for scenarios requiring higher load or multiple geographic locations. The integration aims to streamline performance testing and provide actionable insights into application performance, with the option for scheduled nightly builds for continuous performance monitoring.

Suraj Nath is a software engineer at Grafana Labs, based in Bangalore, who works on enhancing user experience with Grafana Cloud. An avid user of Grafana and Metrictank, Suraj joined the company to contribute to tools appreciated by the developer community. He is also involved in curating failuremodes.dev and has previously contributed to open-source projects like WhsJS, sonic-pi, and PHPA. Outside of work, Suraj enjoys reading, blogging, camping, hiking, and socializing over tea. He recently acquired a graphics tablet for drawing and binge-watched "The Witcher," eagerly awaiting its next season. He humorously wishes for the superpower of knowing the unknown, particularly for navigating traffic.

Loki is a scalable, cost-effective log aggregator designed to streamline operations and enhance team productivity by minimizing technical overhead and organizational friction. Unlike traditional log systems, Loki indexes only metadata rather than full log contents, significantly reducing the need for expensive infrastructure and enabling logs to be stored cheaply in object storage like AWS S3. This cost-effective approach encourages comprehensive logging, crucial for effective incident response without the burden of maintaining complex logging schemas. Loki's unindexed, schema-less design simplifies the operational process, allowing developers and operators to focus on their core tasks without being bogged down by preprocessing pipelines or schema definitions. The use of LogQL, inspired by Prometheus's PromQL, facilitates seamless transitions between logs and metrics, empowering teams to derive insights and manage incidents efficiently. Loki's flexibility and open-source nature make it accessible, scalable from small setups to large clusters, and adaptable across various environments, fostering a collaborative DevOps culture.

Grafana is hosting its inaugural ObservabilityCON from October 26-29, offering a free virtual conference focused on the latest features and use cases of Grafana, Prometheus, Loki, and other tools within the open-source observability ecosystem. Participants can look forward to live talks, demos, technology deep dives, product roadmap insights, and exciting announcements, alongside opportunities for live Q&A sessions with experts. The event caters to observability practitioners of all skill levels and invites attendees to share their own observability stories by submitting proposals to speak. Registration is free, and the conference promises to be a valuable gathering for those interested in the field.

Grafana Labs, a remote-first company, offers a glimpse into the personalized home workspaces of its diverse team, highlighting the unique setups that facilitate both professional tasks and personal passions. Team members employ a range of equipment, from high-end computers and multiple monitors to musical instruments and 3D printers, reflecting their individual needs and interests. For example, Tom Wilkie, VP of Product, uses a basement office with specialized video and audio equipment, while Site Reliability Engineer Chris Trott incorporates musical gear into his workspace. Jeroen Op ’t Eynde prefers a dual-system setup with a Thinkpad and a MacBook, complemented by a variety of peripherals. Other team members, such as Sam Coren and Dave Russell, have optimized their spaces to balance work and creative endeavors, while Justin Hunthrop and Oana Mangiurea emphasize personal touches that make their work environments more enjoyable. This snapshot of Grafana Labs’ workspaces underscores the company's culture of adaptability and creativity, as employees tailor their environments to meet both their professional demands and personal hobbies.

The latest version of the Dynatrace Enterprise plugin for Grafana introduces several enhancements aimed at facilitating the creation of user-friendly dashboards using Dynatrace's Metrics API v2 data and custom metrics. This integration is accessible to Grafana Cloud users and those with a Grafana Enterprise license, allowing them to configure Dynatrace data sources by selecting API types and inputting necessary credentials. The plugin offers a visual query editor with autocomplete and aggregation features, alongside the ability to apply dynamic and static aliases to make metric names more comprehensible. Additionally, users can implement ad hoc filters for efficient dashboard setup without needing to create variables, with further customization possible via a config file. The plugin is designed to enhance observability and is available through a Grafana Cloud account, where users can also access a free trial of Grafana Cloud Pro.

OpenMetrics, a standard for exposing metrics data, is influenced by Prometheus, which revolutionized cloud-native metric monitoring by offering a more modern, efficient alternative to older protocols like SNMP. At KubeCon + CloudNativeCon EU, Grafana Labs' Richard Hartmann and Robust Perception's Brian Brazil discussed OpenMetrics' evolution, its adoption challenges, and its similarities to Prometheus, particularly in format. They emphasized the importance of the _total suffix for counters and the introduction of exemplars to link metrics with traces. Despite political hurdles, major companies such as Google and Uber are interested in adopting OpenMetrics, which aims to maintain a focused approach akin to the Unix philosophy. OpenMetrics largely retains the Prometheus text format while introducing improvements like higher resolution timestamps and the reintroduction of protobuf. It has garnered support from various vendors, including DataDog and OpenTelemetry, and aims for industry standardization with ongoing efforts to formalize specifications and compliance testing. Hartmann remains optimistic about OpenMetrics' potential to standardize Prometheus' capabilities across the broader ecosystem, with future applications likely to integrate metrics, logs, and traces seamlessly.

In a blog post, Goutham Veeramachaneni discusses improvements to backfill methods for importing older data into Prometheus, focusing on two main approaches: retroactive rule persistence and bulk data import from CSV files. Retroactive rule persistence aims to address issues with recording rules in Prometheus, where new recorded metrics only exist from the time a recording rule is added, causing delays in dashboard updates. A community bridge mentee has proposed a design to add a backfill command to promtool, allowing users to specify recording rules and time ranges for backfilling. The bulk import approach involves supporting CSV and OpenMetrics formats for importing data, which could be useful for analyzing data with PromQL or migrating from other systems. A proof of concept for CSV-based import has been developed, but improvements are needed, such as simplifying the CSV format and implementing data batching. Additionally, the post mentions the potential for an online backfill method that would allow streaming data into Prometheus, with plans to refine this based on user feedback and experiences with the offline tools.