March 2019 Summaries

TimeShift's Issue 83 from Grafana Labs highlights updates and articles, including the release of Grafana v6.1.0-beta1, which introduces features like Ad-hoc Filtering for Prometheus and new permissions for dashboard management. The newsletter delves into Grafana's projects and collaborations, such as Loki, a Prometheus-inspired log aggregation project, and its use in companies like Tinder. It covers a range of topics, from scaling metrics with Graphite at Klaviyo to building an air quality monitor using Grafana and InfluxDB. The issue also discusses upcoming events where Grafana Labs will present, including DevOps Days and KubeCon, focusing on innovations in log aggregation and metrics monitoring. Additionally, it provides updates on plugins and encourages community feedback to improve TimeShift.

In the ongoing debate over open-source software licensing, Amazon Web Services (AWS) has been a central figure due to its significant influence and market reach, especially after launching its cloud version of Elasticsearch, initially developed by Elastic N.V. Despite AWS's dominance, its version of Elasticsearch struggled to keep pace with updates from Elastic N.V. and the open-source community, leading to AWS creating a fork called the "open distribution for Elasticsearch" (ODE). This move has raised questions about the nature of open-source contributions and the ethical responsibilities of large companies like AWS in the open-source ecosystem. Meanwhile, several open-source companies, including Elastic N.V. and MongoDB Inc., have altered their licensing strategies to protect their software from being commercialized by major cloud providers without compensation. These changes highlight a broader tension between maintaining open-source ideals and ensuring financial viability, as companies strive to capture the value their projects create. As AWS's actions stir controversy, questions arise about its role and reputation within the open-source community, while Elastic N.V. and others navigate the challenges of balancing openness with commercial interests.

In a blog post by Peter Holmberg, the process of creating Grafana plugins using ReactJS is explored, particularly following Grafana's transition from AngularJS to React starting with version 6.0. This transition is facilitated by the introduction of the @grafana/ui npm package, which, despite being in its Alpha stage and subject to breaking changes, aims to simplify the development of React plugins. The setup involves configuring Grafana to recognize plugins by modifying the custom.ini file to specify the plugins' directory, and structuring the plugins with essential files like plugin.json and module.tsx. The article provides an example of building an RSS panel using React and TypeScript, which involves creating a ReactPanelPlugin class, setting up a panel class with life cycle methods, and employing the rss-to-json library to handle RSS feeds. Editors are used to manage plugin options, and the use of TypeScript types is encouraged to minimize bugs. The build process involves using webpack to convert TypeScript to JavaScript, ensuring compatibility with Grafana's expectations.

Tinder, the popular dating app, has successfully integrated Grafana and Prometheus into its monitoring infrastructure to manage its expansive microservices environment. Originally relying on AWS CloudWatch for metrics monitoring, Tinder transitioned to using Grafana for data visualization and Prometheus for tracking service metrics, due to their scalability and lack of drawbacks found in other solutions. As the app's services grew, Tinder moved to a more scalable setup with individual Prometheus servers for each service and an archive server for long-term data retention. To enhance operational efficiency, Tinder migrated its infrastructure to a Kubernetes environment, utilizing Prometheus Operator for effective monitoring of applications across different namespaces. The Observability Team further optimized the process by developing customized dashboard automation tools using GrafanaLib, ensuring efficient monitoring across various test environments. This strategic evolution has fortified Tinder's observability capabilities, enabling it to maintain robust service health monitoring as its platform continues to expand.

Prometheus 2.8 introduces significant enhancements to the remote_write API by implementing Write-Ahead Logging (WAL), which aims to protect client metrics from network issues. Previously, the API relied on a small in-memory buffer that could lead to data loss or excessive memory usage if remote endpoints were unreachable. The new approach involves writing transactions to a WAL before committing data to long-term storage, allowing Prometheus to pause and retry sending data without data loss or memory problems. This change ensures more predictable memory usage, as the buffer depends on disk size rather than memory, and optimizes the process by encoding data once for multiple send attempts. Although initially intended as a short project, the implementation required addressing various edge cases involving locking, parallelization, and log file integrity. The update has been well-received, with reduced CPU usage and improved memory predictability, and while some issues with WAL corruption have arisen, solutions are being explored. Feedback is encouraged via GitHub to further refine the update.

The latest issue of TimeShift by Grafana Labs covers updates, plugin releases, and event highlights related to Grafana, a popular open-source monitoring tool. The release of Grafana v6.0.2 addresses several bug fixes, including improvements in alert panel links and error handling in dashboards. The blog discusses ongoing changes in open-source software licensing and features insights from GrafanaCon regarding new projects like Loki, a log aggregation system inspired by Prometheus. Tutorials on using Grafana with various platforms, such as Azure Sentinel and VMware Enterprise PKS, are provided, along with information on upcoming events like KubeCon and webinars on open-source monitoring solutions. New and updated plugins, such as the Flowcharting Panel and enhancements to existing panels, are announced, emphasizing Grafana's focus on expanding its functionalities and community engagement.

Booking.com, a travel e-commerce giant, manages an enormous volume of metrics using Graphite, an open-source monitoring tool, to handle millions of metrics per second. Initially employing a stack designed in 2014, the company faced challenges such as single points of failure and inefficient data distribution, which they resolved by rewriting parts of the system, including a relay in C and transitioning to the Go programming language for better performance and maintainability. The company adopted the jump hashing algorithm to improve metric distribution and moved to using Kafka for data transmission, while also integrating Grafana for enhanced data visualization. Despite these advancements, Booking.com continues to refine its system with plans to replace the Whisper storage format due to its inefficiencies and explore new protocols like gRPC for better compression and performance. Smirnov, a former Booking.com employee, highlights the stack's adaptability and its usage by other companies, emphasizing the simplicity and scalability of their setup involving Grafana, Carbon API, and Go Carbon for efficient metric handling and visualization.

The text discusses the evolving landscape of commercial open-source software and its tension with public cloud providers, particularly focusing on the rise of companies like Red Hat and Elastic NV, and their interactions with giants like Amazon Web Services (AWS). Red Hat's pioneering role in commercializing open source set the stage for future companies, transforming open source into a valuable market, as seen with IBM's $36 billion acquisition of Red Hat. As the software industry shifted from physical media to digital services, a new model emerged, treating software as a utility to be consumed and paid for as needed. AWS capitalized on this by offering popular open-source software as a service without compensating original creators, intensifying the conflict with commercial open-source companies like Elastic NV, which adopted "open core" strategies to balance open-source principles with value capture, ultimately leading to a competitive coexistence and innovation in software delivery. The narrative hints at continuing tensions and a potential escalation in the licensing dispute between these entities.

GrafanaCon 2019 in Los Angeles attracted over 350 participants and featured more than 40 talks, focusing on topics like cloud-native observability and real-time analytics in IoT. The event marked the launch of Grafana 6.0, which introduced new features such as the Explore mode for ad-hoc data exploration, integration with the log aggregation system Grafana Loki, React plugin support, and compatibility with major cloud monitoring services. The conference also highlighted Grafana Labs' vision for LGTM, a composable observability stack combining logging, graphing, tracing, and metrics, emphasizing flexibility for users. Interest in Kubernetes, Prometheus, Graphite, and MetricTank was notably high, with Grafana Labs committing to further investments in these areas. Social networking played a significant role in the event, with the v6.0 launch party fostering connections among attendees.

TimeShift's latest edition brings an array of updates and insights for Grafana users and enthusiasts. It highlights the availability of videos and presentations from GrafanaCon LA and announces the release of Grafana v6.0.1, which includes numerous bug fixes and enhancements, such as improvements in dashboard functionality, security, and image rendering. The issue also details Grafana's involvement with the Linux Foundation's newly launched CommunityBridge platform, designed to support open-source communities. Additionally, it covers the introduction of Oracle Cloud Infrastructure as a data source and the benefits of Grafana's integration with OCI. A range of tutorials and articles are shared, including those on setting up IoT monitoring and leveraging Grafana for Kubernetes logging with Loki. The newsletter also promotes upcoming events and meetups related to Grafana, and highlights the ongoing recruitment for Grafana Labs, encouraging engagement with their community through feedback and social media.

New City Energy, led by Andrew Rodgers, has implemented Grafana to enhance sustainability efforts in Washington D.C.'s municipal buildings, aiming for carbon neutrality by 2050. By utilizing Grafana to visualize key metrics, the platform monitors energy usage across 46 buildings, collecting vast amounts of data to optimize energy efficiency and enhance operational comfort. The initiative, rooted in the BuildSmart DC platform, has saved the city $4 million, largely through data-driven retro commissioning processes. By integrating with the VOLTTRON platform and tools like Prometheus, Logstash, and Elasticsearch, the system aggregates data for public access, promoting transparency and accountability. Rodgers highlights the importance of continuous commissioning and distributed energy resources as future goals, aiming to inspire similar sustainability practices beyond D.C., with a focus on empowering all organizational levels to take ownership of energy management challenges.