August 2022 Summaries

OpenTelemetry is experiencing significant growth and integration within the observability landscape, supported by recent developments across various platforms and programming languages. This includes new SDK distributions for OpenTelemetry Go and support for OpenTelemetry logs, which are now in beta, allowing structured logs to automatically correlate with traces, enhancing observability without the need for extensive rewrites. Honeycomb has integrated OpenTelemetry more deeply into its offerings, including features for easier development-time use and local visualizations in both .NET and Go environments. The collaboration with OpenTelemetry contributors aims to improve configurability and ease of use across SDKs, and the initiative is gaining traction, as evidenced by the increasing number of teams using OpenTelemetry and its strong presence at industry events. Honeycomb plans to release additional SDK distributions for Node.js and Python, finalize OpenTelemetry logs support, and expand protocol support to further lower barriers for entry, particularly for front-end developers.

Dashboards, often criticized as "expensive TV for software engineers," can provide significant insights when used analytically rather than as mere performance indicators. They should function as pulse checks for system health and serve as starting points for deeper investigations. The Honeycomb Terraform provider enables the creation of meaningful dashboards by integrating with Honeycomb's query engine to manage queries, annotations, and boards efficiently. This tool helps developers spin up new dashboards for services, infrastructure, or teams using Terraform modules, allowing for tracking various metrics such as service latency and infrastructure utilization. The approach promotes the contextualization of tracing data, enhancing the understanding of system behavior and improving root cause analyses. By leveraging these capabilities, organizations can maintain vital dashboards that offer comprehensive views of service performance, infrastructure metrics, and the interrelation between them, thereby optimizing their debugging efforts and overall observability strategies.

Service Level Objectives (SLOs) are powerful tools for maintaining user experience by identifying and addressing issues before they become significant, yet they can be daunting for teams unsure how to implement or debug them. Observability enhances the effectiveness of SLOs by utilizing event data to provide detailed insights, helping teams manage alert floods and focus on imminent problems. Traditional time-based SLOs often lack granularity and are hard to debug, whereas event-based SLOs offer a more precise approach by qualifying events based on specific conditions, such as error rates and response times. This method allows teams to ask novel questions without additional instrumentation, facilitating fast problem resolution even for unexpected issues. Unlike conventional monitoring alerts that require predefined conditions, SLOs offer a broader perspective, enabling engineers to uncover and address nuanced user experience issues. This approach was notably effective for Honeycomb, where SLOs detected user-impacting problems that were missed by traditional monitoring. For more comprehensive insights, interested individuals are encouraged to explore further resources such as webinars and related literature.

The post explores the use of OpenTelemetry to monitor unit tests, sending data to Honeycomb for visualization, without requiring full adoption in production applications. It focuses on using xUnit for test execution and outlines methods to enhance test monitoring through tracing, which provides deeper insights into application behavior during tests beyond simple run times. OpenTelemetry can treat a test run as a single trace with individual tests as spans or handle each test as an independent trace, offering flexible monitoring options. The article also addresses .NET-specific tracing nuances and suggests using xUnit's IClassFixture and IAssemblyFixture to manage tracer lifetimes effectively. To implement this setup, developers need to configure OpenTelemetry fixtures, integrate Honeycomb for exporting traces, and add a Test Run ID to spans for improved query capabilities. Ultimately, the post demonstrates how OpenTelemetry can boost visibility into application tests, facilitating performance analysis and parallel execution assessments without significant disruption to existing test setups.

In a blog post originally from 2018 and updated in recent years, Charity Majors and others discuss the evolving roles and expectations of engineering managers within software engineering organizations. The post argues for a balanced approach to power distribution between engineering managers and individual contributors (ICs), emphasizing the need to develop a career path for ICs that does not necessarily lead to management positions. Over time, the landscape has shifted, with many engineering managers transitioning back to IC roles due to the increasing challenges and expectations placed on management positions. The text highlights the need for managers to possess heightened emotional intelligence, create psychological safety, and balance business needs with employee support, all while acknowledging the lack of organizational support for meeting these new demands. It proposes an "Engineering Manager's Bill of Rights" to ensure respect and support for management roles, advocating for transparency, feedback flow, and flexibility in compensation aligned with market trends. The evolving conversation around management roles aims to create more equitable and humane workplaces, recognizing the essential role managers play in fostering team cohesion and employee satisfaction.

The text outlines a step-by-step process for testing the functionality of an OpenTelemetry (OTLP) collector by sending a test trace span to Honeycomb, a platform for analyzing distributed traces. It provides a JSON example of a trace span with necessary fields like service name, library name, span name, and span kind, which users can modify for their testing purposes. Users are guided on how to send the span data to the collector using either curl or Postman and verify its arrival in Honeycomb by conducting a query to locate the test span within the specified dataset. The guide also highlights potential troubleshooting steps if the span is not visible in Honeycomb, such as checking the collector logs for errors or verifying the correct API key and environment settings, offering additional resources for new users to explore Honeycomb's capabilities.

Honeycomb Sandbox offers an interactive, hands-on experience for users to explore Honeycomb's data-enriched user interface through a guided scenario, allowing them to identify, assess, and diagnose issues without needing to sign up. The scenario begins with a common problem—customers unable to process orders due to a timing out orders page—and guides users through analyzing this with tools like heatmaps, which visually represent data for quick issue identification. The BubbleUp workflow further aids in uncovering anomalies by asking complex questions, while Traces provide a detailed visual representation of request journeys to pinpoint specific issues, such as identifying slow events like the getDiscounts function due to a deployment. Users can explore additional features beyond the guided walkthrough, such as team activity histories and raw data views, with a hint system to assist in scenario progression, making it an ideal starting point for those new to Honeycomb or observability concepts.

Observability-driven development is proposed as an evolution of test-driven development, enhancing the coding process by incorporating observability tools like tracing and OpenTelemetry to identify and resolve issues locally before reaching production. This approach gives developers similar capabilities to Site Reliability Engineers (SREs), allowing them to understand system behaviors and causality without the need for attaching debuggers or relying solely on logs. Honeycomb's innovative solution facilitates this process by integrating with existing tools and enabling local visualizations, thus improving the developer experience and code quality. Although adopting observability-driven development may require a cultural shift within organizations, the transition can be eased by demonstrating immediate value, starting with small implementations, and gradually expanding the practice. The methodology is aimed at preventing vulnerabilities before they become critical issues, ultimately bridging the gap between development and operational concerns and ensuring that code behaves as expected in production environments.

The text discusses the dynamics of power between managers and engineers in a workplace, emphasizing the importance of maintaining a balance to ensure engineers remain motivated and engaged. It highlights the natural tendency for power to accumulate with managers due to their roles in decision-making and access to personal information, which can lead to engineers becoming managers for the wrong reasons or losing interest in their work. The author introduces the "Engineer’s Bill of Rights" at Honeycomb, which outlines commitments to engineers, such as the freedom to focus on work, opportunities for leadership development, autonomy in technical decisions, and support for career growth. Engineers are encouraged to make progress, be transparent, build trust, and take responsibility for outcomes while managing their time effectively. The text also mentions that managers have their own set of rights and responsibilities, with further details to be shared in an upcoming article, and suggests a related post by Ben Darfler about engineering levels at Honeycomb for additional context.

Observability engineering teams play a crucial role in integrating and managing third-party software solutions within an organization, focusing on selecting and supporting tools that align with the engineering needs and business goals. These teams, often exemplifying "vendor engineering," act as intermediaries between the engineering organization and external vendors, ensuring seamless integration and standardization across workflows, and providing valuable guidance on tool selection and usage. Their responsibilities extend beyond technical integrations to include building a strong culture of observability, facilitating educational initiatives, and managing costs related to observability tooling. They help teams avoid redundancy, ensure coherence, and foster an environment where developers can effectively instrument and understand their software. Rather than running extensive in-house installations, observability teams prioritize the organization's core differentiators and strive to create a supportive culture that enhances visibility and reduces time spent on troubleshooting.

Google's Core Web Vitals (CWVs) are crucial metrics used to evaluate and rank the performance of mobile websites, focusing on user experience components such as loading speed, interactivity, and visual stability. While these metrics are vital for improving search rankings and user satisfaction, developers often find discrepancies between Google’s CWV reports, which are based on real-world usage data, and their lab-based performance tests. Honeycomb's new guide, "Tracking Core Web Vitals with Honeycomb and Vercel," offers a comprehensive approach to bridging this gap by capturing and analyzing real-world CWV performance with tools like Real User Monitoring (RUM). The guide provides a step-by-step methodology for using Honeycomb and Vercel to debug and pinpoint the root causes of CWV issues, thereby enhancing developers' ability to improve site performance efficiently. Through detailed event data analysis, developers can identify specific conditions contributing to poor CWV scores, leading to quicker diagnosis and resolution of performance issues, making it an invaluable resource for those seeking to optimize their web applications.

Elliott and Reid, two engineers at Honeycomb, share their experiences after completing 90 days at the company, highlighting the welcoming atmosphere and transparent culture that emphasizes prompt feedback and documentation. Elliott, a Product Engineer, contrasts the rapid deployment process at Honeycomb with his previous experience at a large company, where deploying new features could take weeks due to complex bureaucratic processes. Reid, who works on the SRE team, appreciates the emphasis on the social aspects of engineering and the company's proactive approach to incident management, which is a departure from his past experiences at startups where pressure was high and reactive responses were the norm. Both engineers note the enthusiasm for demos and the strong sense of community, which fosters collaboration and innovation. Honeycomb’s commitment to good management and openness has allowed them to unlearn defensive programming habits and embrace a culture where fast and iterative development is encouraged.