September 2019 Summaries

Kay Delaney is a software engineer at Grafana Labs, primarily working on the Explore feature while also occasionally contributing to other parts of the Grafana platform. Based in Galway, Ireland, Kay is an active participant in open-source projects, including small contributions to Slate and DefinitelyTyped. Outside of work, Kay enjoys listening to music, watching films with friends, and playing old video games, and recently acquired a Raspberry Pi 4 to set up as an entertainment hub. Preferring to code with ambient music, Kay finds music without lyrics, such as house or ambient music, helps maintain focus. With a slight preference for spaces over tabs in coding, Kay also enjoys strawberry or pistachio ice cream and slowly watches series like Succession, while expressing a fondness for teleportation or time manipulation powers.

The redesign of the thresholds editor in Grafana aimed to improve user experience by making it easier to add and manage multiple thresholds, especially within the new Gauge panel built in React. The original thresholds control was limited and unclear, prompting the team to explore various solutions, including list-based and horizontal line approaches, to enhance clarity and functionality. Ultimately, the design evolved into a vertical line solution that combined elements from previous iterations, allowing users to visually understand and interact with thresholds more intuitively. This final design removed the need for separate "Min" and "Max" values, opting instead for a base color, and incorporated a "+" button for adding new thresholds. Despite these improvements, the single stat panel still uses the old editor, but ongoing experiments by Grafana Labs may lead to further updates. Users are encouraged to provide feedback on the new design to aid in future enhancements.

In the article, Simon Crute addresses the issue of a broken Grafana dashboard caused by a self-referencing variable, which renders the dashboard inaccessible. To resolve this, two main solutions are proposed: using the Grafana API and leveraging the built-in versioning system. The API approach involves creating an API key, utilizing CURL and JQ commands to export the dashboard's JSON, editing it to remove the faulty variable, and re-importing it into Grafana. The versioning method suggests restoring a previous, stable version of the dashboard. Both methods highlight the utility of technical knowledge in API interactions to manage and troubleshoot Grafana dashboards effectively, offering insights into not only fixing issues but also enhancing overall dashboard management.

Callum Styan is a developer at Grafana Labs based in Vancouver, Canada, where he primarily focuses on working with Cortex and Prometheus, having joined the company after contributing to Prometheus independently. He has made significant contributions to open-source projects, notably rewriting Prometheus remote write to incorporate the Write Ahead Log and enhancing Cortex to support samples from High Availability Prometheus instances. With a background in data communications and network protocols, Callum has also worked on documenting the Android ADB protocol. In his leisure time, he enjoys reading fantasy novels, engaging in activities like leathercraft, and organizing the local Go meetup. Callum prefers listening to music or ambient noise while coding and treasures his Sennheiser HD650 headphones for this purpose. His favorite ice cream flavor is Swiss Orange Chocolate Chip from Elevated Ice Cream in his hometown of Port Townsend, Washington, and he recently binge-watched HBO's "Chernobyl."

GraphQL has gained significant popularity, prompting questions about its compatibility with load testing tools like k6. k6 supports load testing GraphQL services running over HTTP or WebSocket, making it versatile for various protocol environments. GraphQL itself is a language for API querying and manipulation that is transport-layer agnostic, allowing servers to operate over diverse protocols such as HTTP, AMQP, and Websockets. The article illustrates using k6 to load test the GitHub GraphQL API, highlighting the need for a Personal GitHub Token to authenticate and showcasing how to create and send GraphQL queries and mutations using JavaScript. A practical example involves querying the first GitHub issue of the k6 project and adding a reaction to it, demonstrating how k6 scripts can replicate user interactions and validate response data. Overall, the article provides a foundational approach to using k6 for performance testing GraphQL services, with ample flexibility for conducting various performance evaluations.

Cortex, a horizontally scalable version of Prometheus, enhances PromQL query performance by leveraging distributed systems and consistent hashing. Created by Grafana Labs, Cortex integrates the PromQL engine to provide a global view across multiple Prometheus servers, overcoming the cardinality limitation of 100,000 time series per query. The Grafana Labs team introduced a stateless query microservice called the "querier" and incorporated extensive caching, including memcache, to improve query speed and efficiency. By aligning start and end times with the query step and splitting queries by day, Cortex can execute queries on multiple cores, significantly enhancing performance. This approach not only allows for parallel execution of queries but also ensures a fair distribution of resources among users in a multi-tenant system. The enhancements have led to substantial improvements in query speed, reducing the time taken from seconds to milliseconds. Future plans for Cortex involve further parallelization by series, automatic identification of subqueries, and ensuring reliability and ease of deployment.

Metrictank has made significant improvements to its data import process for Grafana Cloud customers, particularly for those needing to migrate data from Graphite. Originally, Metrictank's importer tools were designed to move Whisper data into Cassandra, but with the addition of Bigtable and potential future stores, the tools have been refactored to support multiple storage options. The system uses a reader to handle data conversion and a writer to import data into Metrictank-supported stores. This setup allows for schema conversion, namespace adjustments, and efficient data transmission using gorilla-tsz encoding, thereby saving bandwidth and enhancing the flexibility of future data migrations. These upgrades not only facilitate seamless data migration across different database types but also optimize the use of existing stores with built-in enhancements, making the process more efficient for cluster migrations and other purposes.

The Trade Desk, a demand-side platform, successfully modernized its monitoring system by migrating from a labor-intensive, homegrown solution to Grafana Cloud, a fully managed SaaS metrics platform. Initially facing challenges with scalability and reliability, the company sought to improve its monitoring infrastructure, leading to a partnership with Grafana Labs. This collaboration enabled The Trade Desk to leverage Grafana's data visualization capabilities and Prometheus for metrics and alerting, significantly enhancing query times, reliability, and operational efficiency. The migration not only reduced costs and troubleshooting efforts but also rejuvenated the engineering team's focus, allowing them to concentrate on innovative projects rather than maintaining outdated systems. The transition was met with positive feedback from engineers, marking a successful enhancement in performance and team satisfaction.

Goutham Veeramachaneni is a software engineer at Grafana Labs, mainly focusing on backend engineering projects such as Cortex and Loki, while also contributing to the Prometheus project. Based in Visakhapatnam, India, he integrates his passion for cycling into his life, often engaging in long rides, including a notable 1500 km bike-packing trip across Europe. Goutham enjoys coding with music playing out loud and is currently exploring the series "Star Trek: The Next Generation." His GitHub and Twitter handles are "gouthamve" and "putadent," respectively, showing a personal touch to his online presence.

Grafana's future UI aims to enhance the integration of logs, metrics, and traces within observability infrastructures, maintaining its mission to provide a singular, cohesive interface. David Kaltschmidt, Director of User Experience at Grafana Labs, emphasizes the importance of dashboards being purposeful and immutable, evolving Grafana to make observability elements more accessible and interconnected. Current capabilities include features like data links and Explore, which facilitate seamless transitions between different data sources and maintain query context. Future developments are exploring intelligent dashboard suggestions, improved log and trace integration, and the potential introduction of modals and tabbed views to transform Grafana into a more robust troubleshooting interface. The aspiration is to create a collaborative environment with features like saving tabs and adding comments, ensuring Grafana remains a central hub for data collection while evolving to meet users' needs.

Grafana Labs has been optimizing query performance on its metrics platform by integrating Jaeger for distributed tracing, significantly enhancing query speeds by up to 10 times. Initially challenged by poor query performance at scale, they utilized Prometheus to pinpoint slow services and environments, allowing for targeted investigation and improvements through Jaeger's UI. They faced challenges with load balancing in Jaeger, which were resolved by implementing gRPC changes and using Envoy for better traffic distribution. Additionally, Grafana Labs developed a Jaeger mixin to enhance monitoring, which they have contributed to the community. They are working on future improvements, including implementing exemplars in Prometheus to facilitate direct navigation from dashboards to specific traces and exploring scalable tail-based sampling with OpenTelemetry Service to optimize data storage and tracing efficiency.

Brian Gann, a Senior Software Engineer at Grafana Labs based in New Orleans, Louisiana, primarily focuses on developing and supporting enterprise and open-source plugins for Grafana. In his free time, he enjoys playing the guitar and experimenting with crypto mining. Brian prefers coding with loud music rather than in silence or with headphones and finds writing test cases helps him concentrate. His coffee of choice is a café au lait from Café Du Monde, and he has a penchant for binge-watching shows like "True Blood." He favors using spaces over tabs in coding and wishes for the superhero power of time bending.

Grafana Enterprise 6.3 introduces significant enhancements to plugins, authentication systems, and permissions, aiming to improve user experience and control for large enterprises. It expands data visualization capabilities by integrating premium data sources such as Oracle databases and SaaS providers like DataDog. Authentication improvements include the addition of SAML support and enhanced LDAP integration, allowing continuous synchronization of user roles and permissions. The updated permissions system includes Team Sync, which maps LDAP groups to Grafana teams, and Datasource Permissions, granting precise control over data source access. Grafana Enterprise aims to further enhance these features in future releases, including support for GitLab OAuth and the ability to generate PDF reports from dashboards.

Grafana Labs differentiates Grafana Enterprise from its open-source counterpart by offering features tailored to the needs of mature organizations, aiming to support the open-source community while providing premium services. Grafana Enterprise includes integrations with commercial monitoring tools like Datadog and Splunk, advanced authentication and security features including LDAP synchronization and SAML integration, and dedicated enterprise support with 24/7 service. The decision-making process for Enterprise features involves careful deliberation to ensure that these capabilities are uniquely valuable for enterprise-scale production use and do not undermine the open-source version. For instance, while LDAP support is available in both versions, the Enterprise edition offers enhanced synchronization features beneficial for large-scale user management. This approach allows Grafana Labs to cater to enterprise requirements while continuing to enhance the open-source Grafana for broader community benefits.

Grafana's evolution in user experience (UX) reflects a journey of continuous improvement aimed at making data visualization and dashboard management more intuitive and accessible. Originating as an enhancement to Kibana, Grafana sought to reduce visual clutter and simplify editing processes, focusing on clean, elegant interfaces that prioritize data visibility. The introduction of features like a full-screen edit mode and a more comprehensible query structure in its early versions made Grafana popular, particularly within the Graphite community. Over the years, Grafana has incorporated significant developments, such as a plugin system, a backend in Go, alerting capabilities, a new grid system for flexible design, and enhanced backward compatibility. Version 5 introduced a more polished light theme and version 6 allowed for easy visualization switching without panel removal. Moving forward, Grafana Labs aims to address UX challenges by improving user guidance for beginners, refining the admin experience, and expanding the design of forms and panel options, leveraging a growing team focused on UX innovation.