November 2019 Summaries

Grafana Labs participated in KubeCon, showcasing various talks that highlighted advancements and solutions in monitoring and debugging within the Kubernetes ecosystem. Joe Elliott demonstrated a method for debugging live applications using a Kubernetes-native approach with sidecars, allowing for low-impact profiling without significant changes to nodes. Tom Wilkie discussed techniques to accelerate PromQL queries using Cortex, applicable to Thanos and Prometheus. Björn Rabenstein explained how to incorporate latency into SLO-based alerting within Prometheus, drawing from principles in The Site Reliability Workbook. Goutham Veeramachaneni shared insights on managing Cortex, focusing on capacity planning and query performance, and, along with Ed Welch, explored solutions to the deployment complexities of microservices by using a single binary app. Ganesh Vernekar and Matt Layher provided an introduction to Prometheus, emphasizing its multidimensional data model, powerful query language, and system monitoring integrations, underscoring its advantages over traditional monitoring systems.

At KubeCon + CloudNativeCon, Björn Rabenstein of Grafana Labs discussed the integration of latency into SLO-based alerting, emphasizing its importance in site reliability engineering (SRE) practices. He explained the core principles of SLOs, SLIs, and SLAs, and how these concepts are utilized to set and manage alerting thresholds effectively. By measuring error rates and incorporating latency considerations, Rabenstein highlighted the need for alert systems that can respond to both slow and fast error budget burns. He proposed using a combination of long and short time windows for error rate monitoring, which allows for a balanced and responsive alerting system. Additionally, he emphasized the importance of including latency in SLAs, advocating for a model where slow responses are equated with errors to enhance the user experience and maintain service reliability. The talk also touched on the technical implementation of these concepts at Grafana Labs, using tools like Prometheus and Jsonnet for efficient configuration and monitoring. Rabenstein concluded by stressing the value of simplicity in designing alerting systems while maintaining a focus on meaningful performance metrics.

Joe Elliott, a backend engineer at Grafana Labs, presented at KubeCon + CloudNativeCon 2019 in San Diego, demonstrating techniques for debugging live applications in Kubernetes environments. His methods, which are Linux-specific and framework-agnostic, involve using the Kubernetes API and sidecar containers, providing a non-intrusive way to debug at scale without requiring direct node access. Elliott highlighted the ease of using sidecar containers for tool management, which allows for easy cleanup and avoids node impact. He discussed challenges such as tool compatibility with different kernel versions and addressed solutions like baking tools into containers or mounting from the host. Elliott showcased three Linux debugging tools: perf for CPU profiling, LTTng for static tracepoints, and BCC for dynamic tracing, each serving different debugging purposes. He noted the future potential of using Kubernetes ephemeral containers for debugging, a feature expected in Kubernetes 1.16.

Grafana 6.5 introduces a wide array of enhancements and new features across various components including CloudWatch, Explore, Templating, Azure Monitor, and Docker, among others. Notable updates include the introduction of Ubuntu-based Docker images alongside the default Alpine-based images, providing users more flexibility, and significant improvements to the CloudWatch data source in collaboration with Amazon, such as adopting the GetMetricData API for faster data retrieval at scale. This release also sees enhanced functionalities like dynamic typeahead queries, multi-value template variables, and derived fields for log messages, which allow parts of log entries to be turned into links. Additionally, the Explore feature has been advanced with hover/tooltip support in graphs, time-sync of split views, and a simplified log row details view. Alerting support has been expanded to Azure Application Insights, and new provisioning settings allow saving changes to provisioned dashboards directly from the UI. Other enhancements include improvements in authentication processes, such as mixing auth proxy with Grafana login tokens and session cookies, and refined OAuth role mapping. The release encourages users to explore the upgraded image renderer plugin and provides comprehensive documentation and changelogs for detailed guidance on the new and improved features.

At KubeCon + CloudNativeCon in San Diego, Grafana Labs showcased a demo of an upcoming feature for Grafana that introduces distributed tracing datasources, enhancing observability by integrating tracing with existing metrics and logs. The demo featured a dashboard for a simple three-tier application, where periodic errors in the app tier could be analyzed by seamlessly switching between metrics, logs, and traces. This integration is facilitated by shared telemetry metadata and consistent logging of trace IDs, enabling correlation between metrics and logs. A new experimental Jaeger datasource allows internal linking of trace IDs, while external links from matched patterns in Loki have been available since Grafana v6.5 beta. The aim is to eventually provide native visualization of traces in Grafana, supporting a variety of distributed tracing systems beyond Jaeger. The demo highlighted the use of a Docker compose file for running the components together and invited feedback from users trying out the setup.

Grafana Labs has transitioned from contributing to the Cortex project, a horizontally scalable Prometheus implementation for global metric views and long-term storage, to launching Grafana Mimir. Both Cortex and Thanos, a newer project, aim to solve similar challenges but through different methodologies; Cortex uses a centralized push-based model, while Thanos operates with a fanout query system leveraging existing Prometheus servers. Both provide solutions for global metric visibility, high availability, and long-term storage, with Thanos using object stores and Cortex utilizing a combination of object and NoSQL stores, which impacts cost and performance trade-offs. The projects, both built on the Prometheus codebase and part of the CNCF sandbox, have begun collaborating, with efforts like integrating Cortex's query frontend for Thanos and allowing Cortex to adopt Thanos' storage approach, promising improved scalability and cost efficiency as they continue to evolve.

Loki has reached its 1.0.0 General Availability release, marking its transition from beta and signifying its readiness for stable production use. Over the past year, the project has seen significant development, with contributions from over 137 contributors and more than 1,000 updates. Loki has proven its reliability by handling substantial data volumes internally within Grafana Cloud, prompting its availability for customer use. The release underscores a commitment to maintaining API and feature stability while minimizing breaking changes, and it includes features like loki-canary for early log detection and enhancements for Docker and systemd logging. The platform's benefits are exemplified by a case study on Paytm Insider, which achieved substantial cost savings using Loki.

Paytm Insider, a popular platform for purchasing event tickets in India, faced challenges with its logging and monitoring systems as it scaled, experiencing high costs and inefficiencies during traffic spikes. To address these issues, the DevOps team implemented Loki, a centralized solution integrated with Grafana and Prometheus, which streamlined their logging processes and significantly reduced costs by 75%. The solution allowed for efficient log management and faster debugging, decreasing the average response time for latency issues from 30 minutes to 10 minutes and centralizing alerts, which improved correlation between infrastructure and application performance. The team plans to further optimize costs and functionality, and since deploying Loki, they have experienced stable, uninterrupted service even during heavy traffic events.

KubeCon + CloudNativeCon, the flagship conference of the Cloud Native Computing Foundation, is being held at the San Diego Convention Center, where Grafana Labs team members are participating actively. Throughout the event, several talks are being given by Grafana Labs representatives on topics such as accelerating PromQL queries using Cortex, day two operations for Cortex, cloud native architecture configurations, and integrating latency in SLO-based alerting with Prometheus. Additionally, an introductory session on Prometheus and a live demonstration on debugging applications using Kubernetes are scheduled. Grafana Labs also has a presence in the Sponsor Showcase, where attendees can engage with team members to learn about their latest projects and offerings, including Grafana Enterprise and Grafana Cloud.

Robert Milan, a software engineer at Grafana Labs, is involved in working on the Hosted Metrics Graphite platform, with a focus on the Metrictank system and supporting infrastructure, including Kafka and Cassandra upgrades. He contributes to several open-source projects such as Metrictank, carbon-relay-ng, and tsdb-gw. Currently based in Tirana, he often travels between GMT+2 and GMT+8 time zones. In his free time, Robert enjoys media production and running, having recently completed a documentary and working on a series about stereotypes. He is also developing free software for indie filmmakers to facilitate better audio recording. A fan of photography, he likes capturing long exposures at night, prefers coding with music, and has a diverse taste in ice cream, except for durian flavor.

Grafana Labs assists engineers in convincing their companies to adopt Grafana Enterprise or Grafana Cloud by building strong business cases that highlight the value of these tools. Engineers often encounter challenges when seeking management approval, despite proving the technical benefits during trials, such as integrated monitoring capabilities and reduced effort in creating custom metrics. To secure buy-in, Grafana Labs recommends quantifying the solution's time and cost savings, demonstrating personal and organizational benefits, and finding internal allies to support the proposal. Effective presentations, particularly showcasing Grafana's dashboards, can significantly influence decision-makers. Grafana's sales representatives also play a crucial role in navigating company politics, preparing presentations, and providing resources to substantiate the tool's advantages.

PromCon 2019 in Munich featured significant participation from Grafana Labs, where members delivered numerous talks, including four in the main track and six lightning talks, showcasing their expertise in various areas like Cortex and Thanos, Prometheus and Jaeger integration, Prometheus' TSDB improvements, and the evolution of Prometheus histograms. The event emphasized community engagement, with Grafana Labs highlighting their commitment to sustainability by offsetting travel and food carbon emissions. PromCon's single-track format and inclusion of lightning talks fostered a casual and conversational atmosphere, allowing for diverse speaker participation. Attendees were encouraged to continue engaging with Grafana Labs at future events like KubeCon in San Diego.

Grafana Labs will be participating in AWS re:Invent 2022, a major conference held in Las Vegas from December 2-6, featuring a wide array of events such as boot camps, labs, talks, workshops, and keynotes. With nearly 50,000 attendees expected, Grafana Labs will have a presence at Booth 1607 at the Expo in the Venetian Hotel, where visitors can learn about their open observability platform. Key members of Grafana Labs' leadership, including CEO Raj Dutt and other executives, will be available for discussions throughout the week. Additionally, Grafana Labs is sponsoring the closing party at the adjacent IFX conference on December 5, providing a chance for attendees to enjoy BBQ and networking opportunities.

Dieter Plaetinck, a principal software engineer at Grafana Labs, has played a pivotal role in the company's development and growth since its inception. Initially joining forces with co-founder Torkel, Dieter has been instrumental in building open-source monitoring solutions, particularly focusing on MetricTank and Grafana Cloud's evolution. His responsibilities have expanded to include product management for Hosted Metrics Graphite, team leadership, and hiring management, while he still finds time to contribute code and review pull requests. Based in Malta, Dieter frequently travels to Cyprus, Greece, Belgium, and New York, and enjoys hiking and mountain biking in the southwestern U.S. Although he finds noise highly distracting, he has developed strategies to maintain focus during complex coding tasks. Despite his aversion to ice cream, he has a penchant for coconut-flavored products.

At PromCon, organizers announced a commitment to making the event not just carbon neutral but 100% carbon negative, thanks to Grafana Labs' donation. Instead of traditional conference swag like T-shirts, Grafana Labs purchased carbon credits from Cool Effect to offset 465 tons of carbon dioxide, covering 200% of the estimated travel and food emissions for all attendees. The funds from these carbon credits will support the protection of Brazil's Jacunda Forest Reserve. Attendees received carbon-offset acknowledgment cards printed on plantable basil-seed paper, reflecting a shift towards more sustainable conference practices. Grafana Labs intends to implement similar environmental initiatives at future conferences.

Ryan McKinley, VP of Applications at Grafana Labs, shared insights at the InfluxDays conference on integrating real-time data into Grafana dashboards, drawing from his experiences at Natel Energy and a residential solar company. He highlighted the challenges and solutions in streaming sensor data, such as using roll-ups with Influx and integrating custom components in Grafana dashboards. McKinley explored how to manage inexpensive devices by streaming real-time data while maintaining aggregate data in the cloud, leveraging Grafana's evolving query interface for continuous updates. He introduced features like the "From Time" data source to enhance query flexibility and emphasized the importance of efficiently managing data across multiple panels without redundant queries. McKinley expressed a focus on adapting Grafana for industrial applications, aiming for real-time feedback and minimal steps to integrate with SCADA systems, as he transitioned to Grafana Labs to concentrate on industrial and analytic solutions.

Metrictank, a software used in production at Grafana Labs, has introduced the metrics2docs tool to address challenges in documenting metrics for both internal teams and external customers. This tool generates up-to-date documentation directly from Golang code annotations, ensuring that metric descriptions remain current and consistent across both code and public documentation. It supports inline and block comments, allowing developers to document metrics alongside their declarations, which facilitates easy updates. A script is used to automate the generation of configuration and metrics documentation, ensuring all documents remain current. While the team is considering ways to integrate this documentation into Grafana for user accessibility, they also recommend consulting the operations guide for further insights on metrictank's optimal usage. The development of metrics2docs is aided by Go's packages like go/parser and go/ast, which simplify the process of parsing and utilizing Go code.

Ward Bekker shares his process and motivations for creating tutorial videos on Grafana and Grafana Loki for YouTube, which serve both as a method for him to deepen his understanding of the technology and to highlight the work of the Grafana OSS community. As part of his role as a Solutions Engineer at Grafana Labs, he integrates his product demonstrations into video content, employing a detailed production process that includes setting up a demo environment with Docker, drafting a conversational script, and recording voice and screen captures. Bekker uses tools like a Blue Yeti microphone and Apple’s Logic Pro X to ensure high audio quality, while Telestream Screenflow aids in creating and editing the screen recordings. He expresses a desire to continuously improve video production quality and invites audience feedback and suggestions for future content, including potential topics and production enhancements.

Dave Kranowitz, the Vice President of Global Sales at Grafana Labs, brings two decades of sales experience to the role, having previously led enterprise sales for Turbonomic and the U.S. Eastern region for Dynatrace. Based in Barrington, Rhode Island, Kranowitz has a background in 19th-century American history and environmental policy from Connecticut College and enjoys spending time with his family, restoring a 1985 Jeep CJ-7 with his son, and studying Krav Maga. Known for his adventurous spirit, in 1995, he embarked on a memorable cross-country trip, during which he and friends sent postcards from a borrowed cement lawn gnome back to its owner, weaving an imaginative narrative for the gnome's journey. Kranowitz frequently travels for work and relies on his Bose headphones and iPhone 11 as essential gadgets.