March 2022 Summaries

Honeycomb has introduced significant updates to enhance its observability platform by expanding its core data model to include formal Environments and Services, which allows for more precise control over application data. Existing users remain unaffected as they are now part of the Honeycomb Classic team, while new teams created after March 29, 2022, can leverage the new features. These enhancements resolve previous limitations by enabling the creation of Service Datasets that offer both service-specific context and visibility across services within an environment. The update also introduces General Datasets for workloads that don't fit into the Service Datasets, environment-scoped API keys for better management and security, and improved query capabilities across multiple services and datasets. Additionally, markers can now be applied to either a specific service or an entire environment, enhancing event tracking. The update aims to improve organization and segmentation, with an auto-creation of datasets from traces and a limit of 100 datasets per environment, which can be adjusted by contacting support. For Honeycomb Classic users interested in these new capabilities, a migration plan is in development and early adopters are already being onboarded to ensure a smooth transition.

Observability has emerged as a crucial component of modern software architecture, particularly with the rise of complex microservices and cloud-native systems, which necessitate advanced tools to handle high-cardinality, high-dimensionality data. While simpler architectures, such as monolithic applications using traditional LAMP stacks, may not urgently require observability, the insights it offers can still be valuable for understanding the inner workings of a system, diagnosing issues, and improving performance. Observability allows for detailed tracing of requests and resource usage, offering visibility into operations that metrics-based monitoring tools may not provide. The decision to adopt observability tools depends on factors like system stability, the frequency of unexpected issues, and the need for accessible data for engineers, ultimately affecting the quality of life for those who support the system. Even in simpler systems, observability can enhance understanding and provide actionable insights, making it a worthwhile consideration for teams seeking greater visibility and tractability.

Honeycomb, a leading observability platform, has announced its support for the AWS for Games initiative from Amazon Web Services, enhancing its ability to help game developers efficiently manage and improve cloud application performance. As a launch partner, Honeycomb enables game companies to accelerate development and build more reliable services by quickly identifying and addressing complex issues through its purpose-built datastore and query engine. This integration allows developers to gain fast feedback on code performance, ensuring smoother player experiences by leveraging distributed tracing and advanced metrics. Honeycomb's real-time observability capabilities are particularly beneficial for gaming companies with intricate systems, such as CCP Games, which have utilized AWS and Honeycomb to transition from monolithic structures to microservices, significantly enhancing player onboarding experiences. With AWS for Games, developers can access a curated selection of tools and partners tailored to the gaming industry, further supported by AWS's dedicated Game Tech industry specialists and solutions. Honeycomb's collaboration with AWS exemplifies a commitment to providing developers with the necessary resources to create engaging and efficient gaming experiences, with the platform already trusted by notable names like HelloFresh, Slack, and Heroku for rapid incident response and performance optimization.

Honeycomb has announced the 1.0 release of its OpenTelemetry Distribution for Java, marking a significant milestone in its commitment to OpenTelemetry as the standard for application instrumentation. Since declaring its full support for OpenTelemetry in August 2021, Honeycomb has seen a majority of its customers, from startups to large enterprises, adopt this open standard for data transmission. The Java distribution now simplifies the configuration and data export process to Honeycomb, featuring an automatic instrumentation agent, configuration options, and a deterministic head sampler. Additionally, Honeycomb has enhanced its metric handling by introducing automatic compression for application metrics, optimizing them for its pricing model while maintaining data fidelity. This development aligns with the ongoing maturation of OpenTelemetry, which has seen widespread adoption across various programming languages for distributed tracing and metrics, with stability and comprehensive documentation for language SDKs being a primary focus for 2022. Honeycomb remains committed to further improving its OpenTelemetry offerings and contributing to the project's ecosystem to ensure robust support for application observability.

In instrumenting a Node.js application with OpenTelemetry, the choice between `startSpan` and `startActiveSpan` is crucial depending on the desired span hierarchy and expressiveness. `startSpan` is simpler and measures the duration of a task without creating any child spans, meaning it shares the same parent with any concurrent spans. Conversely, `startActiveSpan` requires passing a callback function, which allows the created span to act as a parent to any subsequent spans generated during its execution. This approach provides a clearer hierarchical structure in tracing tools like Honeycomb, where the relationships between activities, such as checking a cache or executing an HTTP GET, are visually represented. Although `startActiveSpan` involves more complexity due to the callback requirement, it offers a more detailed and organized trace, reflecting the causal relationships between tasks and spans.

Dashboards, while popular and useful tools for software engineers, can often be over-relied upon, potentially overshadowing more suitable approaches, especially during incidents. They serve as a window into the system by helping engineers develop mental models to infer system states from limited metrics, but this reliance can become brittle as systems change and disrupt established patterns. During high-pressure incidents, dashboards can become overwhelming due to the sheer volume of data, making them less effective for quick decision-making as they require interpretation of signals that may not always provide clear causal clues. Instead, dashboards should be streamlined to highlight high-level signals, similar to vital signs in a hospital, directing attention to potential issues without diagnosing them outright. The emphasis should be on alerts that guide engineers with relevant context and information, allowing them to bypass extensive data sifting and focus on immediate response actions. This approach suggests that dashboards should complement, not replace, deeper, contextual understanding and other investigative tools in incident response.

Incorporating instrumentation code into an application involves creating a Resource object, which is crucial when setting up the OpenTelemetry SDK for tracing, as it specifies the program sending spans and its execution environment. The Resource object is a key-value store that can hold attributes like service name, instance ID, and version, which are important for identifying the origin of telemetry data. If omitted, the SDK defaults the service name to "unknown_service" with a language suffix. The OpenTelemetry Protocol (OTLP) groups all spans by a single Resource, ensuring that attributes are expanded to every span emitted by the program. Although OpenTelemetry standardizes these attributes for compatibility across telemetry streams, users can bypass creating a Resource by defining the OTEL_SERVICE_NAME environment variable, simplifying the process. The concept of a Resource, originally from the metrics domain, emphasizes the compatibility and standardization that OpenTelemetry strives for between metrics and traces, despite the complexity it introduces in terms of imports and setup.

OpenTelemetry offers a solution to the vendor lock-in problem by providing a standardized instrumentation framework that allows developers to instrument applications once and send telemetry data to any backend system. The text explains how traditional monitoring solutions often lead to vendor lock-in due to their reliance on vendor-specific agents, making it difficult and costly to switch vendors. OpenTelemetry, developed by merging OpenCensus and OpenTracing, introduces an open-source, vendor-neutral framework that eliminates the need for proprietary libraries, giving users the freedom to switch vendors without reinstrumenting their code. The text provides a practical example of using OpenTelemetry with a Node.js Express app to send telemetry data to Honeycomb and New Relic, demonstrating how the OpenTelemetry Collector can be used to send data to multiple vendors simultaneously. This approach empowers organizations to evaluate and compare different vendor solutions without being tied to a single provider, ultimately enhancing their ability to analyze and debug telemetry data effectively.