June 2021 Summaries

Logz.io, founded in 2015, initially aimed to simplify logging with the ELK Stack by offering Elasticsearch and Kibana as a managed cloud service, but soon recognized the need for comprehensive monitoring. To address this, they introduced an Infrastructure Monitoring product based on Prometheus and a Distributed Tracing product based on Jaeger, creating a unified platform for logs, metrics, and traces. Additionally, they integrated a full SIEM to enhance security visibility through log data. Recently, Logz.io revamped its user interface—New Lookz—to streamline navigation and consolidate product features, which is currently in Public Beta for users to try. The new UI promises a more intuitive experience for cloud monitoring and observability, though users have the option to revert to the previous interface if desired.

Game development involves unique challenges compared to other industries, particularly due to the intertwined nature of marketing, development, and release, which creates pressure to meet community expectations and launch milestones. A crucial aspect of managing game projects is implementing effective logging and monitoring pipelines to gain visibility into backend data, which is more critical in gaming than in many other sectors. Different game engines, such as Unity, Blender, GameMaker, Godot, and Unreal Engine, each have distinct logging systems and storage locations, often dependent on the operating system being used. Unity, for instance, categorizes logs into package manager, editor, and player logs, each stored in specific file locations across Windows, Mac, and Linux. Blender allows for detailed logging configurations from the command line, while GameMaker's simpler logging system focuses on a single log file, and Godot offers file logging settings with live monitoring capabilities. Unreal Engine utilizes a file logging analytics provider and Visual Logger for detailed gameplay state reviews. Overall, while this overview introduces logging in popular game engines, it does not delve into applying metrics or traces to development environments, for which services like Logz.io offer robust solutions.

In response to the challenges posed by the COVID-19 pandemic, the OpenObservability community launched a podcast to maintain engagement within the tech community and foster discussions on open source, DevOps, and observability. Over the past year, the podcast has covered diverse topics such as log analytics, metric monitoring, and distributed tracing, featuring contributions from notable figures and projects like Prometheus, Jaeger, and OpenTelemetry. Hosted live on platforms like Twitch and YouTube, the podcast encourages listener interaction and offers an unedited, authentic experience. Popular episodes include discussions on Uber's use of Jaeger, the intricacies of metrics, and the appropriate level of observability for organizations. The podcast's success is attributed to the support from Logz.io and its Developer Relations team, who have helped produce the content and facilitate community engagement. Looking ahead, the podcast plans to explore new topics such as eBPF technology, and invites the community to suggest ideas and participate in future episodes, emphasizing the ongoing evolution of the observability landscape and the importance of continued collaboration and dialogue.

The article provides a comprehensive tutorial on instrumenting a Java application with OpenTelemetry to enable distributed tracing, using a microservices example consisting of two services, Service A and Service B. It details the process of setting up an OpenTelemetry Collector to receive tracing data and export it to a Jaeger backend, specifically Logz.io's managed Jaeger service, eliminating the need for users to deploy their own Jaeger instance. Readers are guided through manual instrumentation steps, including creating spans and injecting span contexts, to trace requests across services and identify system bottlenecks. Additionally, the tutorial highlights the use of the Jaeger UI on Logz.io to visualize the traces, offering insights into application performance. The article also notes the availability of an auto-instrumentation agent for Java applications and provides resources for further exploration of OpenTelemetry instrumentation.

Albert Teoh discusses the integration of distributed tracing in microservices using Istio, focusing on how it simplifies service instrumentation by reducing the amount of code required. Istio's tracing integration can be achieved through either OpenTracing APIs or manual forwarding of Zipkin headers, with the latter being considered simpler. The ongoing discussion in the Istio community about adopting the W3C Trace Context highlights the evolving nature of tracing standards. While OpenTelemetry offers an alternative, its current limitations with B3 header propagation make it less suitable for some use cases. The article provides examples in Go and Java for context propagation, demonstrating both the OpenTracing API and manual methods. Despite the promise of service mesh solutions, some coding is still necessary for trace context propagation, with manual header forwarding offering transparency and simplicity.

Elastic.co's recent move to restrict its Beats open-source shippers from sending data to certain older or non-Elastic distributions of Elasticsearch has sparked concern within the open-source community. This decision, embedded as a breaking change in the minor 7.13 release, forces users to either stay on outdated versions or risk compatibility issues by upgrading. It highlights the challenges of vendor-owned open-source projects and the implications of such changes for users reliant on these technologies. In response, the open-source community, including companies like Logz.io, is turning to alternatives such as Fluentd, Fluent Bit, Telegraf, and Prometheus for log and metric data collection and aggregation. Moreover, the industry is increasingly aligning with OpenTelemetry, a vendor-neutral framework for unified telemetry collection, which promises to mitigate vendor lock-in risks and streamline observability efforts. Logz.io continues to support open-source initiatives, contributing to projects like OpenSearch as part of its commitment to the open-source ethos.

Amitai Stern's blog post explores the process of creating an OpenSearch plugin, specifically a simple REST plugin, to help newcomers gain practical experience and join the OpenSearch community. It delves into the concept of pluggable architecture, which allows developers to enhance a core program's functionality without altering its core components, using the analogy of camera accessories to explain plugin interfaces. The post guides readers through setting up the development environment, implementing the plugin code, and integrating it with OpenSearch, emphasizing the importance of understanding the plugin interface and logic separation. It also covers testing the plugin through YAML tests and running it in OpenSearch, highlighting the learning curve and challenges associated with large-scale projects. The author concludes by reflecting on the confidence and knowledge gained from this experience, encouraging further exploration of OpenSearch's capabilities.

In response to Elastic's decision to relicense Elasticsearch and Kibana under a dual license scheme, AWS and the open-source community initiated the OpenSearch project by forking the last Apache2-licensed versions of these tools. This endeavor involved a meticulous cleanup of proprietary code and branding elements to ensure a pure open-source foundation under the Apache 2.0 license. OpenSearch quickly progressed from its Alpha release in April 2021 to a feature-complete Beta version, with the goal of achieving general availability by July 2021. The project emphasizes a transparent governance model and community involvement, with AWS committing to shared governance and open contributions. With strong initial support and interest from companies looking to replace Elasticsearch, OpenSearch aims to provide a viable open-source alternative while fostering a collaborative community to drive its future development.