July 2020 Summaries

Ecobee's adoption of Honeycomb for observability began organically with engineers trialing the free tier, which eventually led to the enterprise plan as it effectively addressed the company's needs in mapping engineering efforts to user impact. Founded in 2007, Ecobee specializes in smart home devices and organizes its engineering teams in a squad model, promoting shared tools for enhanced customer experience insights. Teams initially used Prometheus and Grafana for monitoring but found limitations in these tools for performance tuning and opted for Honeycomb to gain deeper insights into system latency issues. This switch was driven by grassroots advocacy from engineers like Ray Slakinski, who highlighted the inefficiency of existing tools in resolving API latency spikes. With Honeycomb, Ecobee implemented OpenTelemetry's auto-instrumentation, which seamlessly integrated with their system, providing immediate actionable insights. The company's squads now widely use Honeycomb's features, such as Service Level Objectives (SLO), to align closely with business needs, improve incident response, and optimize performance. Ecobee credits its culture of shared service ownership and collaborative learning for the successful integration and expansion of Honeycomb across its teams, which has become an indispensable tool for operation and decision-making.

Honeycomb's event-based pricing model focuses on the number of events sent, raising the question of how to maintain data quality while reducing event volume. Sampling is a technique to achieve this, allowing users to selectively transmit data without significantly compromising fidelity. Honeycomb supports the use of custom sampling logic, which can be implemented via the Ruby Beeline by overriding the config.sample_hook to filter less valuable or noisy events, such as redundant Redis or SQL commands, using a deterministic sampler. This enables trace-aware sampling by ensuring consistent decisions for all spans within a trace. However, users must be cautious to avoid common pitfalls, such as dropping spans with children or incorrectly implementing head-based and tail-based sampling, which can lead to incomplete trace waterfalls. The platform's configurability allows users to test and refine their sampling strategies, ensuring that only the most relevant data is transmitted while maintaining an accurate representation of their overall dataset.

Observability enhances understanding and interaction with applications in production by providing insights into both live and developmental environments, as discussed by Pierre Tessier in Raw & Real. Honeycomb utilizes observability to improve production systems, notably by using features like Markers to track deployments and identify performance bottlenecks, such as a slow query causing sluggish page load times. By tracing and addressing technical debt, Honeycomb improved load times significantly, demonstrating the value of observability in reducing friction and enhancing team productivity. Observability also transformed Honeycomb's build process by identifying bottlenecks in the test suite and transitioning to a concurrent build pipeline, which dramatically reduced build times and increased efficiency. This data-driven approach has allowed Honeycomb to make impactful changes, optimize resource allocation, and maintain efficient build cycles, as highlighted in their informal and insightful Raw & Real discussions.

Alex Vondrak, a Senior Platform Engineer at true[X], details his experience troubleshooting a Redis-related issue using Honeycomb, which led to the discovery of a workaround. The problem began when clients connected to a new Redis database spiked unexpectedly, leading to connection failures due to a RedisLabs-imposed connection cap. The issue stemmed from a deployed workaround that over-relied on blocking operations, specifically the BLPOP Redis command, which exacerbated connection problems. Despite various attempts like setting client timeouts and monkey-patching libraries, Vondrak struggled to identify the issue's root cause. By employing Honeycomb for live tracing, he discovered that a client-side read timeout was adding unnecessary delays, causing endless retries and idle connections. By modifying his Redis command to circumvent the read timeout logic, he successfully reduced connection counts and restored functionality. The experience highlighted the importance of using live tracing for debugging and ensuring instrumentation in problematic codebases.