March 2020 Summaries

Grafana Labs is transforming Grafana from a dashboarding application into a comprehensive observability platform, aiming to facilitate the correlation of metrics, logs, and traces to expedite issue resolution. During a FOSDEM 2020 presentation, Andrej Ocenas, a full stack developer at Grafana Labs, highlighted Grafana's open-source and open-platform nature, allowing integration with various data sources. The talk elaborated on using Prometheus and Loki for app instrumentation and label configuration to correlate metrics and logs, showcasing how matching labels can streamline querying across these data sources. For tracing, Ocenas discussed the integration of Jaeger and the necessity of logging traceIds, emphasizing future enhancements like direct linking from latency graphs to traces. Grafana Labs is also working on introducing "Experiences" to group related data sources logically, aiming to create seamless connectivity and improve user interfaces by eliminating the need for manual configurations.

In response to the shift towards remote work due to the COVID-19 pandemic, Grafana Labs has shared technical guidance on optimizing video conference calls, a key component of the remote work experience. Key recommendations include always muting when not speaking to minimize background noise interference, using headsets with boom microphones for superior audio quality, and utilizing computer-based calls over phone calls for enhanced audio fidelity. The guide emphasizes the importance of using video to improve communication and advises monitoring bandwidth and latency to ensure seamless connections, suggesting wired connections and WiFi adjustments as potential solutions. Additionally, it recommends testing equipment and network conditions before important calls to identify and rectify any issues.

Jessica Müller, a UX designer at Grafana Labs, is based in Darmstadt, Germany, and focuses on creating human-centered designs by conducting user research and integrating feedback into interface designs. With a background in graphic design, she emphasizes accessibility, usability, and aesthetic appeal in her work, often employing collaborative and interdisciplinary approaches. Outside of her professional life, Jessica is passionate about music, having grown up with a musical background, and now composes, sings, and produces her own albums. She is also a vintage fashion enthusiast, blogger, and enjoys a variety of hobbies including cooking, attending management meetups, and spending time with her family playing board games. Fluent in Japanese and with a talent for Dance Dance Revolution, Jessica's ultimate dream is to live part-time in Japan, a step closer now thanks to her remote job.

Trust is a foundational element in building and maintaining communities, as explored in this blog post by Richard "RichiH" Hartmann. The author delves into the multifaceted nature of trust, emphasizing its role in fostering positive interactions and shared purposes within a community. Trust can be categorized into personal, topical, and contextual dimensions, each contributing to how individuals perceive reliability and authenticity. The post highlights the importance of reliability in maintaining trust, suggesting that following through on promises and transparent communication are key to nurturing long-term trust. Additionally, the ease with which trust can be lost, likened to the burning of money, underscores its fragility and the challenges associated with regaining it once damaged. The post concludes with a reflection on the framing of communication, which can significantly impact perceptions of trustworthiness and the acceptance of future promises or topics.

Grafana Labs, after ceasing contributions to Cortex, has introduced Grafana Mimir, a scalable open-source storage solution for Prometheus. In Cortex and Loki, the ring is a vital component stored in a key-value store like Consul or etcd, used for sharding series and logs among ingesters. However, traditional storage methods face issues with update conflicts and resource consumption. To address these, Grafana Labs has implemented the gossip protocol using the Memberlist library to propagate the ring data, avoiding the need for a central key-value store. This approach reduces network communication, enables frequent timestamp updates without increasing message frequency, and allows more efficient handling of state information. Despite minor discrepancies in the ring's state across nodes due to continuous updates, the gossip protocol provides sufficient reliability for system operations. The adoption of gossip in Cortex and Loki has shown promising improvements in performance and resource utilization, encouraging users to explore this new method and provide feedback.

Grafana's Oracle Enterprise plugin allows users to seamlessly integrate Oracle data and stats into Grafana dashboards, leveraging the GoLang oci8 library to avoid the need for additional data warehousing. This enables the visualization and analysis of both business and system data, such as system health, user activity, and resource utilization, through a unified interface. Users can easily switch between multiple Oracle instances within a single dashboard or aggregate metrics across instances, offering a comprehensive view of database performance. The plugin supports relational and time series data queries, features SQL auto-complete for efficiency, and includes alerting capabilities for monitoring database issues such as blocked users. Installation involves adding the Oracle data source to Grafana's configuration, with dependencies like the Oracle Instant Client required. The Oracle plugin is accessible to Grafana Cloud account holders or those with a Grafana Enterprise license, offering a powerful tool for enhancing observability and data-driven decision-making.

As COVID-19 disrupted global life, Grafana Labs reassured customers of its full operational capacity, highlighting its remote-first structure that minimized the impact of the pandemic on its services. The company emphasized the readiness of its support and service reliability engineering teams, which remain fully staffed and accessible 24/7 to assist with Grafana Enterprise and Grafana Cloud. Grafana Labs also mentioned its recent $24 million funding, ensuring long-term financial stability and continued support for the open-source projects and communities that form the foundation of its business. The communication conveyed a message of solidarity, inviting customers to reach out with any inquiries.

Grafana v6.7 introduces several new features and improvements, including enhanced authorization options with Azure OAuth integration, allowing users and groups to be assigned roles from the Azure Portal. The release adds a Google Sheets plugin and a Stackdriver project selector that enables querying different projects without changing the data source. A new query history feature in beta allows users to manage and reuse previous queries more easily. Grafana Enterprise sees updates like white labeling for customizing application titles, configurable reporting with options for timeout and concurrency, and the ability to export dashboards as PDF documents in either portrait or landscape mode. Additionally, the update enforces a minimum dashboard refresh interval to prevent excessive load on data sources. The release acknowledges community contributions and offers comprehensive information on upgrading and the complete changelog.

Grafana Cloud Agent, now known as Grafana Agent, is a memory-efficient subset of Prometheus designed for hosted metrics, reducing memory usage by up to 40% compared to Prometheus. Unlike Prometheus, it omits querying and local storage, instead focusing on remote_write to platforms like Cortex. This makes it suitable for large-scale deployments by distributing memory load across clusters, particularly in Kubernetes environments, through optional host filtering. While it lacks Prometheus's recording and alerting capabilities, these functions are supported within Cortex and Grafana Cloud, ensuring alert reliability depends on the hosted solution. The Agent is compatible with any platform supporting the Prometheus remote_write API, not just Grafana Cloud. Future enhancements include support for Graphite and Loki, and a new scraping service mode to improve metrics capture from problematic nodes. The introduction of Grafana Cloud Agent addresses scalability challenges while maintaining a Prometheus-like experience, providing a streamlined alternative to existing collection agents.

Navigating remote work while managing children at home presents unique challenges, particularly as schools and childcare facilities close, necessitating parents to juggle work, homeschooling, and childcare simultaneously. Grafana Labs offers insights into effectively managing this dynamic by emphasizing open communication and renegotiation of needs and expectations with family members and employers. Establishing a structured schedule that mimics school routines helps create a predictable environment, while empowering children to be more independent can alleviate parental stress. It's vital to balance work commitments with family needs, adapting to disruptions and being transparent with colleagues about potential delays. Implementing practical tools, such as noise-cancelling headphones and creative activities, can facilitate a productive home environment. Maintaining a supportive and loving atmosphere is crucial, as understanding the emotional undertones of this adjustment period is key to fostering resilience and cooperation within the family.

Peter Štibraný, a Slovak expat living in Denmark, works as a software engineer at Grafana Labs, focusing on projects like Cortex, Loki, and Prometheus. Recently appointed as a Cortex maintainer, Peter is dedicated to enhancing the blocks storage in Cortex for improved operation and cost efficiency. In his free time, he enjoys playing the harmonica and studying music theory. He values the global team at Grafana Labs and the opportunity to work on projects that are widely appreciated. Despite the challenges of remote work, he finds fulfillment in his role and is passionate about his hobbies, like building robots with his daughter and practicing harmonica. He aspires to slow down time to accomplish more and dreams of playing music with others and performing with a symphonic orchestra.

As the COVID-19 pandemic ushered many into remote work, Grafana Labs shared valuable insights from its remote-first team to help others transition effectively. Key recommendations include allowing time to adjust to the new work setting and establishing a fixed schedule to maintain work-life balance, alongside setting clear boundaries to prevent work from encroaching on personal time. Dressing for work, maintaining a virtual version of a commute, and creating a designated workspace can signal the start and end of the workday. Effective communication is crucial, with an emphasis on documenting processes and using video calls to stay connected. Social interaction remains important, and team members are encouraged to engage in virtual social activities. Additionally, they highlight the importance of investing in a proper home office setup and establishing house rules with family members to align personal and work environments. Balancing household chores with work, managing snacking habits, and ensuring regular outdoor activity are also advised to maintain productivity and well-being. Ultimately, individuals are encouraged to find personalized strategies that best suit their unique circumstances and evolve with changing needs.

Tanka is a Jsonnet-based tool developed by Grafana Labs to improve the deployment and management of Kubernetes infrastructure, offering a more flexible and efficient alternative to YAML. At FOSDEM 2020, Grafana Labs engineers Tom Braack and Malcolm Holmes demonstrated how Tanka simplifies deploying Grafana instances by reducing the complexity and repetition associated with YAML configurations. Jsonnet's features, such as functions, patches, imports, and packages, enable more concise and adaptable configurations, allowing users to manage development and production environments without duplicating code. Tanka supports the creation of shared Jsonnet libraries, facilitating the use of pre-built modules and mixins for various applications, which enhances scalability and collaboration. During the presentation, Holmes provided a practical demo showcasing the streamlined deployment process using Tanka, significantly reducing the time and code required compared to traditional methods. Interested users are encouraged to explore Tanka further through an online tutorial and engage with the Grafana community.

The blog post, part of the "Learn Grafana" series, delves into the use of dual axis graphs in Grafana, highlighting their utility in comparing trends between time series of different magnitudes or units through a second Y-axis. It explains the process of creating dual axis graphs by assigning time series data to the right Y-axis, allowing for a clearer visualization of correlations, such as between CPU load and response time or traffic and latency. Despite their benefits, dual axis graphs pose risks due to potential misinterpretation, as different scales can manufacture false correlations. To mitigate these risks, the post advises on best practices like using contrasting visualization options and aligning baselines to ensure clarity and prevent misleading representations. The series also offers additional resources for enhancing Grafana usage, such as dynamic dashboards and optimizing load time.

GrafanaCon 2020 is set to take place in Amsterdam on May 13-14, with a final decision regarding its schedule to be made by March 31 due to the COVID-19 pandemic. The conference will feature sessions on topics such as Grafana 7.0, the future of Prometheus, updates on Metrictank 1.0, Grafana integrations, series lineage, metadata, and new features in Loki, as well as Tanka and dashboards as code. Early-bird registration is available until March 27, offering a significant discount, and registrants will receive a full refund if the event is canceled or postponed.

Owen Diehl, a software engineer at Grafana Labs based in Washington, D.C., contributes significantly to the open-source projects Cortex and Loki, which assist in monitoring applications through metrics and logs. In his role, he focuses on optimizing query processes for large datasets by employing techniques like caching and query rewriting to enhance parallel processing capabilities. As a maintainer of the Loki project, Owen has recently worked on the query frontend component and extended the LogQL query language, with efforts in query sharding that originated in Cortex but now also support Loki. Outside of work, Owen enjoys hiking, playing video games, and running, and he finds self-direction and creative problem-solving to be the most rewarding aspects of his job at Grafana Labs. In his leisure time, he also looks forward to starting a garden and is a fan of crawfish, particularly the New Orleans style, while also interested in trying the Vietnamese variety from Houston.

In the first part of a series on online communication, Richard "RichiH" Hartmann discusses the challenges and nuances of written communication in the open-source community, emphasizing the importance of assuming good faith to avoid misunderstandings. The lack of social cues in written exchanges can lead to misinterpretations that escalate into disputes. Hartmann, drawing on his experience with freenode, notes that misunderstandings often remain hidden until they become disputes, and suggests that knowing each other well through in-person meetings or video calls can help provide context to online interactions. Cultural differences, such as varying customs around greetings, further complicate communication, and Hartmann advises that creating shared documents for persisting information can uncover potential points of contention. He also highlights the value of social calls and informal interactions in fostering better understanding, while noting that the next entry in the series will focus on effective question-asking techniques.

Monitoring client certificate expirations can be challenging, but Grafana Labs offers a solution that leverages Grafana and Prometheus to make certificate status visible and observable. Joe Elliott describes how he designed the cert-exporter tool to expose information about certificates that are not supported by the standard Prometheus blackbox exporter, such as x509 certificates on disk, certificates embedded in kubeconfig files, and those stored in Kubernetes secrets. This tool is particularly beneficial for tracking certificates in kops-based Kubernetes clusters, allowing administrators to manage certificate rotations proactively and avoid outages. A Grafana dashboard complements cert-exporter by visually representing certificate statuses, highlighting those nearing expiration, and displaying an error counter, ensuring administrators can address potential issues promptly.

Observability, as discussed by Grafana Labs cofounder and CEO Raj Dutt, is a crucial approach for understanding and monitoring complex software systems, diverging from traditional IT operations monitoring by integrating data analytics into system management. During a podcast with Dan Woods, Dutt emphasized that observability involves the real-time correlation of metrics, logs, and traces to empower developers to deploy applications safely and efficiently, enhancing both operational and cultural facets within organizations. By utilizing platforms like Grafana, companies can consolidate diverse data sources and leverage open-source ecosystems without being confined to monolithic data storage solutions. Dutt highlighted the importance of observability in cost savings through effective infrastructure management and auto-scaling capabilities, and the potential role of AI in predictive analytics to further enhance system understanding. He also addressed the challenges and opportunities in open-source software, noting the balance between value creation and capture, especially in the context of enterprise software infrastructure and the evolving relationship with cloud vendors.

The article provides an extensive review of various open-source load testing tools, comparing their performance, developer user experience, and suitability for automated testing. It examines tools like ApacheBench, Artillery, Drill, Gatling, Hey, JMeter, k6, Locust, Siege, Tsung, Vegeta, and Wrk, highlighting features such as scripting capabilities, multithreading, and distributed load generation. The author, with a disclosed bias towards k6, details each tool's history, development status, and specific use cases while conducting performance benchmarks to assess their efficiency in generating traffic and measuring response times accurately. The review reveals significant performance variations among the tools, noting that while Wrk excels in raw traffic generation, k6 offers a balanced experience for developers with its strong scripting API. Additionally, it observes improvements in tools like Locust, which has enhanced its performance, while others like JMeter and Siege have seen a decline. The article concludes with personal preferences for different tools based on specific testing needs, such as simplicity, scripting language preference, and traffic generation capabilities.

Grafana 6.6 introduces a new feature that allows administrators to enforce a minimum interval between alert rule evaluations, addressing a longstanding request to manage backend load caused by overly frequent user-defined alerts. Implemented by @Uepoch, this feature sets a default alert evaluation interval of one minute, which can be adjusted by users, but administrators now have the ability to set a global minimum using the new configuration option, alerting.min_interval_seconds. This ensures that any alert rule with a frequency lower than the specified minimum will trigger a warning and be evaluated at the administrator’s set interval instead. The change applies to both new and existing alert rules, meaning that if an administrator imposes a higher global interval after a rule's creation, the user will be warned of the override. The Grafana v6.6 stable release also includes improvements in various areas such as Alerting, CloudWatch, Explore, Loki, Graphite, and Stackdriver.