February 2020 Summaries

Gracie Sharpe is a People Ops Coordinator at Grafana Labs, based in New York, where she administratively assists the People Ops team and manages the high-volume recruitment process. She joined Grafana Labs to enhance her HR skills and was immediately impressed by her manager during the hiring process. The most rewarding aspect of her job is guiding candidates through the recruiting process and onboarding them. Outside of work, Gracie enjoys making music and hiking, and she holds a B.A. in HR and Psychology, which she completed in three years. She has a unique talent for opera singing, having trained for over 14 years, and humorously notes her ability to eat an entire pizza by herself.

SkySilk Cloud Services employs Grafana dashboards to effectively monitor and analyze data sources, allowing for quick response to alerts and issues in real-time. Utilizing this open-source tool, SkySilk maintains a comprehensive view of regional system health, employing real-time analytics and a user-friendly interface to monitor both internal infrastructure and user uptime. Integrated with Icinga Web 2, Grafana aids in troubleshooting by providing network-related insights and alerts, allowing for efficient information access and problem resolution. Emphasizing simplicity and community, SkySilk supports open-source technology, offering a one-click Grafana deployment template to simplify analytics setup, thereby enabling users to focus on development. SkySilk actively encourages its diverse community of developers, businesses, gamers, and educators to share their projects and engage with the platform, fostering a collaborative environment.

Grafana dashboards can greatly benefit from being configured as code, a concept explored in a FOSDEM 2020 talk by Malcolm Holmes and Julien Pivotto. They introduced Grafonnet, a Jsonnet library, as an efficient tool for generating and managing Grafana dashboards through code, allowing for consistent and scalable dashboard creation. The talk highlighted the challenges of maintaining numerous dashboards with varying user preferences and emphasized the advantages of using dashboards as code to ensure uniformity and easy updates. Jsonnet, a JSON superset, was showcased for its ability to simplify JSON generation with advanced features like functions, patches, and imports, enabling collaborative and flexible dashboard management. Additionally, Holmes and Pivotto demonstrated the creation of reusable dashboards and panels, underscoring the potential for widespread code reuse across teams. Looking to the future, there are discussions on expanding Grafana's capabilities with native Jsonnet functionality to further integrate dashboards with existing code and version control systems, enhancing collaboration and deployment efficiency.

Ivana Huckova's guide offers a comprehensive walkthrough on setting up Prometheus Alertmanager to send alert notifications to Slack, PagerDuty, and Gmail. The setup process begins with creating alerting rules in Prometheus to define the conditions under which alerts are triggered, such as when an instance goes down. These rules are then linked to Alertmanager, which manages the alerts by grouping them, muting notifications, and selecting the appropriate receivers. For Slack integration, users need to configure a Slack workspace with Incoming WebHooks, and add the webhook URL to the alertmanager.yml file. PagerDuty integration requires creating a service and using an integration key for notifications, while Gmail alerts necessitate setting up an App Password for secure email transmission. The guide emphasizes utilizing Grafana Cloud for a streamlined experience and highlights the ease with which Alertmanager can manage and deliver alerts across different platforms, enhancing the monitoring of IT systems.

The blog post by Malcolm Holmes explains how to connect Prometheus-Ksonnet to Grafana Cloud, offering a solution for monitoring large clusters by overcoming limitations of local installations. Grafana Cloud Hosted Prometheus provides long-term storage, supports multiple clusters, and ensures high availability, making it suitable for advanced monitoring needs. The post includes step-by-step instructions for setting up a Grafana Cloud account, creating a Hosted Metrics Instance, and configuring Prometheus with the necessary URL, user ID, and API key for integration. By adding remote write configurations and external labels, users can distinguish and manage metrics from different clusters, enabling seamless querying of both local and cloud-hosted Prometheus data. The guide emphasizes the ease of managing metrics across multiple clusters and the benefits of using Grafana Cloud for enhanced monitoring capabilities.

Christine Wang, a Solutions Engineer at Grafana Labs, is based near Los Angeles and is known for bringing a personal touch to technical integrations. She made her first open source contribution by enhancing the Dark Reader Chrome extension and is active on GitHub as christineywang, though she prefers to stay off most social media platforms. In her spare time, Christine enjoys activities like writing and hosting dinner parties, but she is most content watching shows like "90 Day Fiance" while enjoying Indian takeout. She codes best in silence, adapting easily to various environments due to her upbringing, and has a penchant for matcha ice cream with blackberries. Christine recently fulfilled her wish for a fully remote job by joining Grafana Labs, and she humorously acknowledges her handyman skills as recognized by her son.

Being a technical writer at Grafana Labs can be unexpectedly adventurous, as illustrated by the author's experience with testing Grafana installation documentation on Windows and Linux. While attempting to install a plugin on both operating systems, the author encountered permission issues on Windows that were resolved by running PowerShell as an Administrator, highlighting a gap in the documentation. Further challenges arose when the author discovered a Linux command mistakenly included in the Windows instructions and undocumented global options in the Grafana CLI. These experiences led to updates in the documentation, emphasizing the dynamic nature of technical writing which involves constant problem-solving and collaboration with developers. The author concludes by inviting suggestions for further documentation improvements, showcasing the ongoing journey of refining technical content.

Loki, a log storage solution built on the Cortex architecture, has implemented optimizations to significantly reduce its log storage requirements, achieving nearly three times less data storage than previous versions. Originally, Cortex introduced changes like label sorting and using Memcached to avoid duplicate chunk uploads, which were then applied to Loki. Loki, which utilizes log lines instead of metric samples, adapted these improvements with a focus on chunk size rather than a fixed time window due to its distinct data structure. By synchronizing chunk cutting based on both time and utilization thresholds, Loki reduced duplicate data storage from three copies to approximately 1.2 copies, enhancing storage efficiency and search speed. Despite this progress, some issues persist, such as synchronization challenges among ingesters and occasional desynchronization. The ongoing development aims to fine-tune the synchronization process to further enhance storage efficiency, particularly for busy series that would benefit most from deduplication.

Eldin Nikocevic and Christine Wang from Grafana Labs' Solutions Engineering team introduce the Splunk plugin for Grafana, which enables users to integrate Splunk's log management capabilities with Grafana's data visualization tools. This plugin allows for a comprehensive, single-pane view of system metrics, facilitating quicker correlation and debugging for reduced mean time to recovery (MTTR). Users can configure Splunk as a data source, utilizing both raw and visual query modes, and take advantage of advanced options for enhanced search performance and data source permissions. The plugin is available for Grafana Cloud account holders or those with a Grafana Enterprise license, and it supports the visualization of logs alongside other data sources. For those seeking alternatives, Grafana Loki is suggested as an open-source, cost-effective logs aggregator. The post underscores the importance of correlating logs with metrics for effective incident response and encourages users to explore Grafana Cloud with a free trial.

Florian Boucault, a software engineer at Grafana Labs based in Paris, contributes primarily to the Metrictank cluster, which is compatible with Graphite, and focuses on improving reliability and performance by working with Golang, Python, and shell scripts. His role involves site reliability engineering, managing internal automation tools, and enhancing Metrictank, especially in terms of data transfer between clusters. Florian engages in open-source projects, notably Metrictank, and enjoys personal activities such as spending time with his daughter and advocating for environmental protection. He prefers coding with music and recently binge-watched "Borgen," appreciating its portrayal of Danish politics. Florian has a hidden talent for lock-picking, adding a unique skill to his profile.

Grafana Labs is organizing its inaugural user group meetup in the Bay Area on Thursday, February 27, at Strava HQ in San Francisco. The event will provide a hands-on experience and cover key aspects of observability, including metrics, logs, and tracing. It will begin with a two-hour roundtable discussion on Prometheus and Grafana, featuring insights from Grafana Labs engineers and experienced users. This will be followed by a presentation on the future of observability at Grafana Labs and breakout sessions on topics such as logging best practices and scaling Prometheus. The meetup will conclude with a happy hour, offering attendees the chance to network with Grafanistas and fellow users. Those interested in attending are encouraged to sign up.

Dieter Plaetinck highlights a tool called "memo" that streamlines the process of adding annotations to Grafana, a platform for visualizing metrics, logs, and other data. Annotations in Grafana are useful for overlaying events onto data charts to correlate disparate pieces of information, such as marketing events or configuration deployments. While Grafana's API allows for the creation and management of annotations, not all processes are automated, necessitating a manual method for adding annotations in certain cases. The memo tool addresses this by integrating with Slack and the shell, allowing users to easily convert Slack messages into Grafana annotations with contextual tags for efficient categorization and retrieval. This tool enhances the ability to share and document events across different domains, accommodating both automated and manual inputs to maintain comprehensive data insights.

Grafana Labs, after years of leading Cortex development, has transitioned to using Grafana Mimir for scalable metric storage, but continues to utilize Hashicorp's Consul for service discovery and data distribution. Although Consul's KV store was initially useful, scaling operations revealed limitations, prompting Grafana Labs to implement unconventional solutions like compacting every 128 versions and using memory disks to optimize performance. These adjustments allowed for greater scalability and consistency, albeit at the cost of reliability, by operating with a single-node Consul setup. To manage API load, rate limits were introduced, and the team plans to incorporate the gossip protocol to further reduce dependency on Consul. Despite these "abuses," Grafana effectively adapted Consul to meet its operational needs.

Malcolm Holmes provides a detailed guide on setting up a monitoring system using Tanka and the prometheus-ksonnet library within a Kubernetes environment. The process involves leveraging Jsonnet for enhanced collaboration and ease of configuration, allowing users to install Prometheus and Grafana along with the kubernetes-mixin library, which offers best practice dashboards and alerts for Kubernetes monitoring. The guide assumes a pre-existing Kubernetes cluster and provides step-by-step instructions for installing necessary tools like Tanka and Jsonnet Bundler, downloading the prometheus-ksonnet library, and configuring it within the Kubernetes context. Users can then apply these configurations to deploy Grafana, Prometheus, and NodeExporter, and access the services locally. Holmes also references additional resources for further exploration and mentions an upcoming blog post on integrating the setup with Grafana Cloud.

Malcolm Holmes, a software engineer at Grafana Labs based in Stroud, U.K., is a member of the Infrastructure team responsible for maintaining and developing Kubernetes infrastructure for the company's development teams. In addition to his work at Grafana Labs, Malcolm contributes to open-source projects, including Tanka and has a history with the Apache Software Foundation. Outside of work, he enjoys teaching meditation courses, designing with OpenSCAD for his 3D printer, and recently started learning Kung Fu. His GitHub handle is @malcolmholmes, although he does not use Twitter. Malcolm's personal interests include developing a web application for his father and exploring creative projects with his 3D printer, such as making custom brackets and Grafana-themed items. His unique achievements include developing a meditation approach for parents, while his leisure activities involve binge-watching series like "The Mentalist."

DeveloperWeek 2020 is set to take place next week in the Bay Area, attracting approximately 8,000 attendees for various workshops, open houses, and conferences. Grafana Labs will participate in the event, with team members available at Booth #504 during the DeveloperWeek Expo at the Oakland Convention Center on February 13 and 14. The team will be present to discuss their open-source projects like Grafana, Loki, and Metrictank, as well as their Enterprise and Cloud products. Attendees have the opportunity to meet with key team members, including VP of Applications Ryan McKinley, Solutions Engineer Christine Wang, and Enterprise Account Executives Sam Leavitt and Sean Saunders, and can also schedule a meeting through a contact form.

Loki v1.3.0 introduces a significant new feature in the form of a query frontend, designed to enhance query processing efficiency and protect against denial of service attacks. This component can be integrated seamlessly into the existing Loki architecture, allowing it to split and parallelize queries by time to improve processing speed significantly, especially beneficial for handling high-throughput log streams. Additionally, the query frontend employs per-tenant query scheduling to ensure fair distribution of resources, mitigating the risk posed by problematic tenants overloading the system. Looking forward, the Loki team aims to implement result caching, inspired by Cortex, to store and reuse entire query results efficiently, and query sharding, which transforms queries into more parallelizable forms for enhanced performance.

PromQL, the querying language for Prometheus, is designed to extract and manipulate time-series data from metrics collected and stored in Prometheus's time series database. Prometheus employs three primary data types: scalars, instant vectors, and range vectors, each serving different needs for data representation and manipulation. Labels in Prometheus enhance metric flexibility by allowing users to distinguish metrics with shared names but different attributes, though they can lead to unexpectedly large query results. Prometheus supports four metric types—Counter, Gauge, Histogram, and Summary—each with distinct conventions and uses, such as monitoring cumulative values, fluctuating metrics, or distribution of values. Functions like delta, increase, and rate, alongside aggregation and binary operators, enable complex data analysis and visualization, particularly when used with tools like Grafana. Despite its strengths, PromQL's syntax can be non-intuitive and requires careful attention to detail to ensure queries align with user intentions, as illustrated by the author's personal exploration and recommendations for further reading on Prometheus's documentation.

Grafana Labs has introduced a new ServiceNow data source plugin for Grafana, allowing users to integrate and visualize data from the ServiceNow platform alongside other metric data sources within Grafana dashboards. This plugin supports ServiceNow's Incident Management and Change Management solutions, enabling teams to gain deeper insights into IT service management by correlating incident data with broader system metrics. Users can set up real-time dashboards to reduce incident response times and configure alerts to prevent SLA breaches, as well as generate daily PDF reports for management. The plugin is accessible to those with a Grafana Cloud account or a Grafana Enterprise license, and it encourages users to leverage features like data links and the News Panel for enhanced dashboard functionality. Grafana Cloud offers a free 14-day trial for new users to explore these capabilities.