November 2019 Summaries

The text reflects on the spirit of gratitude, emphasizing the importance of practicing thankfulness beyond Thanksgiving, highlighting its psychological and physical benefits. It discusses Honeycomb's approach to fostering a culture of gratitude through regular acknowledgments in company meetings, and its commitment to creating accessible and efficient software monitoring tools for engineers. The narrative then delves into the observability community's achievements, focusing on collaboration and the development of a blameless culture that encourages learning from incidents without finger-pointing. It also highlights the significance of open-source contributions, particularly the OpenTelemetry project, which aims to unify distributed tracing, metrics, and logging. The text concludes by celebrating the sharing of observability journeys by users, which fosters community growth and knowledge sharing, inviting readers to join in this journey while wishing them a happy Thanksgiving.

On November 6, 2019, a slow memory leak in a company's ingest backend systems led to the intermittent rejection of 1-3% of customer telemetry data over several short periods, causing widespread backend crashes and request failures. Despite quickly detecting the issue through Service Level Objective (SLO) measurements, confirmation bias and unclear incident handling delayed resolution, as engineers mistakenly attributed the problem to AWS's Application Load Balancers. The company later identified the true cause as a memory leak and process restarts, promptly reverting a problematic commit to stabilize the service. The incident highlighted the need for improvements in incident declaration, assumption questioning, and communication protocols, ultimately emphasizing the importance of transparency and iterative process refinement to prevent future occurrences.

Modern applications increasingly rely on distributed microservices, necessitating a unified approach to telemetry data collection. OpenTelemetry, an open-source observability framework, has emerged as the industry standard, consolidating past efforts like OpenTracing and OpenCensus to provide comprehensive support for traces, metrics, and logs in a vendor-neutral manner. Managed by the Cloud Native Computing Foundation, OpenTelemetry offers language-specific instrumentation libraries, APIs, SDKs, and the OpenTelemetry Collector to facilitate seamless telemetry data generation, processing, and exportation. While OpenTracing, a precursor focused solely on tracing, is now in maintenance mode, OpenTelemetry incorporates its most valuable features, along with advanced capabilities like auto-instrumentation and a universal protocol, making it the preferred choice for organizations aiming to futureproof their observability infrastructure. Migration from OpenTracing to OpenTelemetry can be executed incrementally, with support for both frameworks in systems like Honeycomb, ensuring a smooth transition to the more comprehensive OpenTelemetry ecosystem.

In a live panel discussion hosted by Marcus Blankenship for Gitprime, Charity Majors, Ian Nowland, and Rich Archibold explore the complexities and challenges of becoming a manager of engineering managers, emphasizing the unpredictable career paths that led them to their current roles. The panelists discuss the nuances of managing seasoned professionals, the distinction between managing a single team versus managing multiple layers of leadership, and the importance of giving effective feedback. Ian Nowland highlights the significance of relationship capital in leadership roles, while Rich Archibold shares practical strategies and a formula for successful management. Charity Majors provides insights into identifying potential successful managers. The discussion is designed to be an informative resource for anyone aspiring to move up the management ladder or evaluate their current managers, with the webinar available on-demand for further learning.