January 2020 Summaries

Richard 'RichiH' Hartmann, the Director of Community at Grafana Labs, shares insights about his role and personal interests in a self-interview format. He is responsible for fostering interactions with various Grafana Labs communities, including Prometheus, and organizes events like PromCon while handling behind-the-scenes tasks that contribute to successful software projects. Currently based in Munich, Germany, Hartmann has a strong connection to Prometheus, where he engages in community work and event organization. He decided to work at Grafana Labs due to the shared vision and the opportunity to influence various industries positively. Outside of work, he enjoys activities like reading, cooking, and woodworking, and has a penchant for unique ice cream flavors and tech gadgets like noise-canceling headsets. Hartmann also recounts instances of heroism, such as saving children from drowning and resolving SCUBA accidents, which he considers among his greatest achievements.

Grafana Enterprise, the commercial version of the popular open-source software Grafana, is increasingly being adopted by large organizations due to its ability to consolidate data into a "single pane of glass," offering a unified view that enhances efficiency and decision-making across vast engineering teams. This transition from data silos to a cohesive platform allows companies to identify and resolve incidents faster, saving significant costs incurred from downtime. Grafana Enterprise provides companies with advanced features such as secure data access, expert support with service-level agreements, and the ability to create customized dashboards, which are particularly valuable for those bundling Grafana into their own products. Additionally, it supports a comprehensive observability approach, integrating metrics, logs, and traces from various systems, in line with Grafana Labs' strategy to offer an open and composable observability platform. The ease of transitioning from the open-source version to Grafana Enterprise, without disrupting existing configurations, further supports its appeal among large enterprises seeking to enhance data accessibility and collaboration.

Grafana Labs eagerly anticipates attending FOSDEM, an annual open-source conference in Brussels, as it provides a platform for idea exchange, feedback, and collaboration within the community, and allows remote Grafana team members to meet in person. The company will maintain a stand at the conference, marking its fourth year of participation, where attendees can engage with Grafana projects such as Prometheus and Loki, among others, and enjoy complimentary stickers. A notable feature of the event is the Monitoring and Observability Devroom, co-organized by Carl Bergquist and colleagues, which will host talks on emerging topics like tracing, with a particularly strong lineup this year. Several Grafana Labs team members will deliver presentations, including topics on Grafana's capabilities and Tanka, a tool for Kubernetes infrastructure management.

84.51°, a data analytics and marketing company owned by The Kroger Company, faced challenges with decentralized metrics and communication between its Technology and Product teams, leading to discrepancies in data reporting. Erin O'Brien, a Lead Service Manager, spearheaded the initiative to centralize metrics using Grafana Enterprise, enabling a more holistic and efficient approach to data visualization. The introduction of Grafana dashboards facilitated better alignment and understanding of metrics across different teams, allowing for streamlined data collection and reporting. This initiative received positive feedback from company leadership, who appreciated the one-stop-shop solution for visualizing all relevant data, which ultimately enhanced decision-making processes. The company plans to conduct metrics workshops and integrate Grafana with other systems like Service Now to further standardize and improve their metrics reporting. With Grafana, 84.51° is now able to present a cohesive story from its data, helping various teams to focus on relevant metrics and aiding business leaders in making informed decisions.

Grafana 6.6 introduces a range of new features and improvements across various components such as Alerting, Explore, Loki, and CloudWatch. Key updates include the introduction of a new Stat panel designed to replace the Singlestat panel, enhanced alerting configurations to reduce backend load, and design updates to the TimePicker for improved usability. The release also adds support for custom data units, allowing users to personalize unit displays on panels, and introduces white labeling options in the enterprise version for greater customization. Additional enhancements in Explore include log message line wrapping, unique log labels, and a context tooltip for better data query refinement. The update also includes changes to cookie management to align with Chrome 80's SameSite attribute modifications and improvements in CloudWatch for automatic period calculation based on time range, reducing the risk of exceeding data point limits. The release encourages users to upgrade to the Enterprise Edition for seamless updates and highlights ongoing efforts to transition from AngularJS to React, which may affect plugin behavior.

Joe Elliott is a Senior Software Engineer at Grafana Labs, based in Louisville, Kentucky, where he specializes in managing Jaeger trace infrastructure and collaborates with open-source tracing communities. He actively contributes to various open-source projects, including OpenTelemetry, Loki, and Kubernetes, and maintains a presence on GitHub under the handle joe-elliott, though he is less active on Twitter as @actually_chores. In his free time, Joe enjoys reading, raising his children, and occasionally playing video games, although he is not particularly interested in gadgets beyond the utility of a modern phone. Joe is adaptable to different coding environments, having grown up in a noisy household, and finds that diving into typing helps him get into the coding zone, followed by a thorough review and editing process.

Grafana Labs experienced a significant 23-hour outage affecting their Grafana Cloud Graphite service due to a Google Cloud Platform incident that degraded the performance of GCP Solid State Disks, impacting Cassandra clusters, a critical part of their infrastructure. This led to a cascade of issues, including failed write operations to Cassandra and connectivity problems with Metrictank instances. The outage required a complex recovery process involving updating Kubernetes StatefulSet configurations and restarting Metrictank instances, compounded by a longstanding deadlock bug in the Metrictank code. The incident prompted several lessons and improvements, including fixing old bugs, enhancing recovery procedures, and emphasizing the importance of global teamwork and effective monitoring. Despite the challenges, the team managed to mitigate long-term data loss, and the experience drove them to make their systems more resilient for future incidents.

Loki 1.3.0 introduces significant performance improvements and new features, particularly benefiting those using Loki in microservices mode with the new Loki Query Frontend, which facilitates query sharding based on time for parallel execution. Previous releases, 1.1.0 and 1.2.0, addressed API incompatibilities with Prometheus and improved the hashing algorithm to reduce fingerprint hash collisions and out-of-order errors. The 1.3.0 update includes enhancements such as a configurable compression algorithm, with Snappy chosen for its speed despite less efficiency in storage, and a target chunk size configuration to optimize storage and querying overhead. Additionally, a new mechanism for syncing chunks across ingesters reduces log data redundancy, and Promtail has been upgraded to support syslog messages, broadening its functionality beyond file tailing. Further details and updates about Loki's future developments are anticipated in upcoming blog posts.

Cortex, a scalable Prometheus implementation and CNCF project, is poised for significant growth despite Grafana Labs' shift in focus to Grafana Mimir for long-term storage solutions. While Cortex has been praised for its ability to handle large-scale data and provide a centralized monitoring experience for enterprises, it faces challenges such as reliance on NoSQL stores and a need for improved documentation and user experience. Efforts are underway to eliminate the NoSQL dependency by integrating object storage solutions, facilitating interoperability with Thanos, and enhancing user onboarding through better documentation and UI improvements. Cortex's enterprise appeal is underscored by its ability to centrally manage metrics and alerts, exemplified by its use at companies like GoJek. The goal for 2020 is to enhance Cortex's adoption by addressing these challenges and exploring potential synergies with Thanos, ultimately driving broader enterprise adoption of Prometheus.

Diana Sarlinska is a UX Designer at Grafana Labs, based in Kraków, Poland, where she focuses on understanding user needs and incorporating them into product designs. Her role involves conducting user research and gathering feedback to improve the user experience, and she primarily contributes to the open-source project Grafana. Outside of work, Diana enjoys biking, swimming, reading non-fiction and graphic novels, and socializing with friends. She has a unique talent for remembering numbers and mimicking accents, despite not being able to replicate melodies. Diana also recently checked off a bucket list item by traveling around Iceland with a tent.

Cortex, originally developed as Project Frankenstein by Julius Volz and Tom Wilkie, is an open-source project designed to enhance the scalability and reliability of Prometheus by providing multi-tenant, horizontally scalable, and long-term storage capabilities. Initially supported by Grafana Labs, which later transitioned its focus to Grafana Mimir, Cortex has evolved to handle the ingestion of hundreds of billions of samples per day through innovations like consistent hashing, replication, and schema optimization. These improvements address bottlenecks in data compression and indexing, allowing efficient storage and retrieval of metrics across distributed systems. The architecture employs techniques like index caching and query parallelization to manage vast volumes of data and improve query performance. Currently, Cortex supports various storage backends, including Bigtable, Cassandra, DynamoDB, GCS, and S3, and is being refined for use in enterprise environments, emphasizing centralized monitoring and alerting capabilities. Future developments aim to integrate TSDB with Cortex, leveraging Thanos to optimize performance and maintain efficient data management across multi-tenant systems.

GitLab's integration of Grafana and Prometheus into its Omnibus package enhances its monitoring capabilities by providing users with pre-installed, ready-to-use dashboards for visualizing metrics. This development emerged from the need to manage numerous metrics efficiently across GitLab's software and applications, particularly after the integration of GitLab.com. Prometheus was initially bundled into the Omnibus installation for monitoring, and later, Grafana was added to facilitate data visualization without requiring manual installation. This integration supports both GitLab's service-oriented architecture and individual app monitoring in Kubernetes, offering a comprehensive monitoring solution. GitLab's transparency ethos is reflected in its public Grafana dashboards, and while users can theoretically add custom plugins, the default setup is designed to be functional and effective. Despite some dashboards facing issues due to slow queries or restricted data access, the integration is largely complete and aims to make monitoring an accessible feature for developers, with ongoing efforts to inform users about these available tools.

Diana Sarlinska, a UX Designer at Grafana Labs, shares her experiences as a mentor at Google Launchpad, a global acceleration program that supports startups by connecting them with experts from diverse backgrounds. Her role involves providing guidance on UX/UI and product development, drawing on her experience with complex tech products. She highlights the collaborative and dynamic nature of these mentoring sessions, which include workshops, keynotes, and one-on-one discussions to address the unique challenges faced by startups in various industries. The recent Launchpad event in Tallinn brought together startups from multiple countries and mentors from different fields to explore issues like medical trial recruitment and demand-driven pricing. The program not only offers a platform for startups to gain valuable insights but also serves as a learning and networking opportunity for mentors, fostering a sense of community and shared learning.

Migrating a Grafana configuration database from PostgreSQL to MySQL/MariaDB can be streamlined by following a series of steps, starting with creating an empty MySQL database and adjusting the Grafana configuration to connect to it. The process involves exporting tables from the PostgreSQL database and using a series of SED commands to ensure compatibility with MySQL, such as removing PostgreSQL-specific statements and adjusting schema notations. After exporting the data to a file, it can be imported into MySQL, allowing Grafana to use the new database setup upon restarting. This approach requires certain assumptions for simplicity, such as both databases running on the same server without security constraints, but adjustments may be necessary in production environments.

Marco Pracucci, a software engineer at Grafana Labs, is known for his contributions to the Prometheus, Cortex, and Loki teams, with a focus on making Cortex scalable both on-premise and in the cloud. Based in a small town near Rimini, Italy, Marco spends his time between the Cortex and Thanos communities, working on a new storage engine for Cortex that utilizes Prometheus TSDB blocks, which is expected to enhance scalability and reduce operational costs. Despite his busy schedule with a full-time job and two young children, Marco enjoys woodworking in his spare time, recently building a pinball machine, and he values the quality of ingredients in ice cream over specific flavors. Active on Twitter as “pracucci” and open to direct messages, he recently added noise-canceling headphones to his toolkit to maintain productivity in less-than-quiet environments and recommends "Umbrella Academy" as a binge-worthy show despite not typically being a fan of superhero genres.

Grafana Labs introduced Tanka, an open-source tool designed to improve the process of deploying to Kubernetes by leveraging the Jsonnet language. Unlike YAML, which is static and often cumbersome for managing complex configurations, Jsonnet offers dynamic capabilities such as abstraction and conditional logic, making it more suitable for Kubernetes deployments. Tanka emerged as a response to the limitations of existing tools like Helm and the discontinued ksonnet project, providing a streamlined workflow that includes show, diff, and apply commands without imposing additional conceptual overhead. It focuses on environment awareness by tying configurations to specific namespaces within clusters and ensures outputs are at least valid JSON, reducing logical errors. Tanka remains unobtrusive by allowing users to import existing YAML files and simply converting Jsonnet to YAML for Kubernetes. The project is actively developed with contributions encouraged, and future enhancements are planned to address issues such as better packaging support and garbage collection.

Joey Bartolomeo's discussion on lock-free observations for Prometheus histograms highlights a challenge in concurrent programming with Go, particularly when updating Prometheus histograms, which are bucketed counters. Bjoern Rabenstein, an engineer at Grafana Labs, presented at GopherCon UK 2019 about handling updates to these histograms in a concurrency-safe manner using atomic operations instead of Go's communicating sequential processes (CSP) or mutexes, due to the inefficiencies these methods introduce in highly concurrent scenarios. Rabenstein explains that Prometheus histograms require updating multiple metrics simultaneously, which can lead to data races if not handled correctly. He outlines his approach to solving this problem by leveraging atomic operations to increment numbers safely without locks, even though this method is complex and can lead to misalignments on certain platforms, such as 32-bit systems. Rabenstein's solution involved ensuring 64-bit alignment by adjusting the data structure layout to use pointers, thus maintaining performance while ensuring safe concurrent operations. His work underscores the need for careful consideration of concurrency issues in widely-used open-source projects, as even minor inconsistencies can be significant when the software is deployed globally.

During a lightning talk at PromCon EU, Grafana Labs developer Callum Styan discussed significant improvements to Prometheus' remote write feature, the culmination of a six-month effort. Originally, remote write had issues with buffering unsent samples in memory, leading to data loss if the buffer filled up, which could crash the system (OOMkill). The new approach involves using Prometheus' existing write-ahead log (WAL), which records all necessary data, thus creating an on-disk buffer that prevents memory overflow and data loss by not reading more data if the buffer is full. This change means remote write now uses more memory under normal conditions but avoids crashing in worst-case scenarios. Additional work continues, including improving sharding and other enhancements to refine the system further.

The blog post explores the quest for the most efficient hashing method for data distribution within Metrictank, a clustered time-series database. It explains how Metrictank utilizes partition IDs to distribute data across cluster instances, thereby ensuring even load distribution and resilience to instance failures. The text investigates various hashing functions, such as FNV-1a, SipHash, xxHash, and MetroHash, with a focus on the jump hash function, which requires data to be preprocessed into a uint64 format for optimal performance. The experiments reveal that xxHash combined with jump hash provides a superior balance of computational efficiency and even data distribution, leading Metrictank to adopt it as the default for setups with tag support. The study highlights the ability of the new setup to process up to 10 million metrics per second on a single core, with additional benefits such as reduced bandwidth and disk space usage due to the elimination of key publication in Kafka messages.