September 2021 Summaries

Honeycomb has introduced CONCURRENCY, a new feature in their Query Builder, which provides insights into the simultaneous execution of processes within AWS Lambda, aiming to solve the mystery of varying query execution times. The feature was developed in response to challenges in understanding Lambda's behavior, particularly in a co-tenancy environment where multiple customers share resources. By visualizing the concurrency of Lambda invocations, users can better understand the delays and inefficiencies in job scheduling, such as the impact of the retry mechanism triggered by AWS's HTTP 429 responses when no instances are available. The CONCURRENCY aggregate allows users to see not only when processes start but also how long they run, offering a clearer picture of the system's state and revealing inefficiencies in resource allocation. This tool makes it easier to identify bottlenecks and optimize resource use, providing a significant advantage over previous methods that lacked such detailed visibility. Honeycomb encourages users to explore this feature and share their experiences to enhance its utility further.

The text discusses the evolving concept of observability in the tech industry, contrasting it with traditional monitoring and emphasizing its importance in understanding complex systems. Observability is described not just as a buzzword but as a crucial capability that allows engineers to analyze and debug systems by examining outputs without prior knowledge. The author criticizes the oversimplification and mislabeling of observability as merely a set of metrics, logs, and traces, arguing that such interpretations miss its true value. Highlighting the industry's shift towards microservices and the resulting complexity, the text underscores the need for meaningful technical terms and practices that address unknown-unknowns in modern systems, advocating for a clearer understanding of observability's capabilities to prevent misapplication and to guide effective problem-solving.

Sampling is a method used to extract a subset of data from a larger dataset to make inferences about the whole, and while it is not flawless, it can be highly effective in managing large volumes of complex event data when implemented using Honeycomb’s trace-aware sampling proxy, Refinery. GOAT, an e-commerce platform specializing in designer sneakers and apparel, exemplifies the need for an efficient sampling solution due to its high volume of customer-facing requests. At the 2021 hnycon, Kevan Carstensen, a Backend Engineer at GOAT, shared that their small team relies on sampling with Refinery to manage this data volume, cut through noise, and resolve issues efficiently. However, Kevan emphasized that sampling is not universally suitable and should not be the default choice without understanding its nuances and potential drawbacks, such as increased cognitive load and maintenance requirements. GOAT’s implementation of Refinery involved integrating it into their internal Platform as a Service (PaaS) and tuning it to meet their specific load requirements, leading to enhanced visibility and cost management. The process highlighted the importance of rules-based sampling to optimize event quotas and budgets, allowing GOAT to maintain a stable infrastructure while focusing on insights provided by the sampling data.

Honeycomb Metrics, now generally available to Enterprise customers, offers a modern approach to metrics by integrating system-level metrics with event-based observability practices in a single interface, allowing for seamless debugging and issue identification. This service converts time series metrics data into events, stored in a columnar data store, and provides customizable visualizations to correlate system issues with application performance problems. Honeycomb emphasizes a simplified, predictable pricing model without additional fees beyond standard event-based volume pricing. The platform supports data from various providers, such as AWS CloudWatch and Prometheus, and encourages users to leverage metrics for system insights while using events to understand application behavior. Existing Enterprise users have immediate access, while Free or Pro users can trial the service, with support available through live tours and Q&A sessions.

Honeycomb has launched its new Honeycomb Metrics feature for enterprise customers, enhancing its observability platform by integrating metrics capabilities to quickly pinpoint and resolve system issues. This development enables engineering teams to gain comprehensive visibility into both application behavior and the health of underlying systems without needing to switch between tools, thus streamlining debugging workflows and reducing downtime. By combining event-driven observability with system-level metrics, Honeycomb offers a holistic view of the user experience, allowing developers to diagnose performance and quality issues more efficiently. The Honeycomb Metrics feature supports data from sources like OpenTelemetry, Prometheus, and Amazon CloudWatch, creating visualizations that help correlate system impacts on application performance. This integration not only reduces cognitive load by minimizing context switching but also enhances the speed and accuracy of issue identification and resolution. The initiative reflects Honeycomb's commitment to improving business performance by providing robust observability solutions for modern development teams.