April 2022 Summaries

In the context of code instrumentation, Honeycomb offers a distinct approach compared to traditional metrics-based tools by utilizing structured data blobs, or events, which allows developers to append multiple dimensions to events without additional cost. This model reduces the need to delete instrumentation, as doing so is typically driven by cost constraints in metrics-based systems where high cardinality metrics can be prohibitively expensive. Honeycomb charges by the event rather than by the metric, making it economical to retain a rich set of dimensions, which can be beneficial for future debugging and analysis. The platform supports up to 2,000 dimensions per event, with a cap of 10,000 per dataset, providing ample space to capture diverse context while maintaining safeguards against errors. Honeycomb also offers an interactive demo and a free tier for users to explore its capabilities without immediate commitment.

Observability Driven Development is transforming software engineering culture by emphasizing the importance of robust instrumentation and data richness for effective system monitoring and troubleshooting. It encourages teams to move beyond traditional logging and monitoring by adopting practices that enable a comprehensive understanding of system health and performance, using tools like OpenTelemetry to standardize telemetry data. Honeycomb supports this transition by providing flexible data querying capabilities and encouraging the use of wide events and high cardinality attributes to enhance investigatory processes. By applying the scientific method to system investigations, teams can identify and address gaps in telemetry, improve system resilience, and test incremental changes safely. Furthermore, Honeycomb extends observability to CI/CD pipelines with features like buildevents and markers, allowing teams to visualize build performance and track deployment impacts, ultimately fostering a more robust and reliable production environment.

Miss O11y introduces Martin Thwaites as a new band member at Honeycomb and delves into the concept of "Baggage" in OpenTelemetry (OTel), which is crucial for passing contextual information between spans in distributed systems. Baggage ensures that context values are standardized across platforms and frameworks, allowing applications to read and use them regardless of language, which is vital in massively distributed systems. Although Baggage offers a systemic way to carry data like Account Identification or User IDs across service boundaries, it poses risks if sensitive information is included, as it propagates widely without inherent integrity checks. The text highlights the importance of using Baggage judiciously to avoid performance issues and data leaks, illustrating potential pitfalls with an anecdote where Baggage data inadvertently affected Honeycomb's tracing systems. Proper use of Baggage requires explicitly appending it to span attributes, a process simplified by tools like the BaggageSpanProcessor in Honeycomb's OpenTelemetry libraries, but caution is advised to prevent unintended consequences.

UX designers are grappling with feelings of burnout and marginalization despite being recognized as key players in business success, largely due to a misunderstanding of their process and role within organizations. While design thinking has gained traction, many companies fail to integrate designers into strategic conversations, treating design as a mere aesthetic function rather than a collaborative and iterative process critical for solving complex problems. This disconnect leads to unrealistic expectations placed on designers, who often work without adequate support or leadership, and are expected to establish design culture and manage UX debt single-handedly. To bridge this gap, designers must learn to articulate the business value of their work and foster relationships across departments, while business leaders should strive to understand and support the unique processes involved in design work. By recognizing the long-term value of a creative design culture, companies can better align expectations and resources, ultimately enhancing both product success and designer satisfaction.

Tracing provides a more comprehensive and connected view of software execution than traditional logging by offering insights into connection, performance, concurrency, and causality across services. While logs have been the original tool for debugging, tracing through spans allows for a clearer expression of what happens during execution by providing timestamps, durations, attributes, and causal connections between spans. This interconnected view helps in understanding the order and impact of events, which is especially beneficial in complex distributed systems. The use of tools like OpenTelemetry for autoinstrumentation makes implementing tracing less code-intensive than maintaining detailed logs. While logs can be sufficient for sequential, single-service applications, modern observability standards, which emphasize clarity and connection, make tracing a preferred method for gaining insights into software performance and behavior.

Vlad Ionescu, an independent consultant and technology enthusiast, shares his experiences benchmarking container scaling on AWS, particularly focusing on the performance of ECS on Fargate Windows containers. He emphasizes that true observability in tech systems is about enabling teams to ask insightful questions rather than just relying on logs, metrics, and traces. Through testing, Ionescu discovers the performance bottlenecks related to the download times of container images, especially the complexity of Windows container images due to their non-distributable base artifacts. He highlights how using Honeycomb's derived columns feature allowed him to efficiently analyze data without additional costs, leading to significant performance improvements by setting Docker daemon flags for private-use images. This exploration underscores the importance of observability in empowering engineers to optimize systems, with Ionescu using Honeycomb's free tier to facilitate his discoveries.

OnCallogy Sessions are a strategic initiative designed to improve the experience of being on-call by fostering a collaborative and supportive environment for discussing on-call practices and challenges. Held weekly on Mondays, these optional sessions provide a platform for participants to share information, address uncertainties, and propose improvements in on-call procedures. The sessions vary in format, including operational presentations, hands-on sessions, planning discussions, and free discussions, encouraging participation regardless of whether attendees are currently on-call. Through these sessions, the organization aims to cultivate a culture of continuous learning and enhancement of on-call operations, ensuring psychological safety and open communication. The initiative emphasizes the importance of structured time dedicated to refining the on-call experience, making it a habit to discuss and address issues collectively, and ultimately striving to make the on-call process less stressful and more efficient.

Miss O11y addresses the confusion around the use of three-letter acronyms (TLAs) in the context of Honeycomb's offerings, specifically focusing on the Service Level Objectives (SLOs) available in the Honeycomb Pro plan. SLOs are a method for setting and measuring uptime and reliability goals, a practice that originated from the Apollo lunar missions and was adapted for software by Google's Site Reliability Engineering (SRE) team. SREs apply engineering rigor to software operations to ensure reliable service delivery. The importance of SLOs lies in their ability to provide quantifiable goals for service reliability, which helps teams determine the balance between feature work and reliability improvements. Honeycomb's platform allows teams to define, measure, and proactively manage SLOs using existing traces and telemetry data, offering insights into customer impact and potential outages. This proactive approach helps teams address issues before they affect users significantly, minimizing false alarms and improving the reliability of software delivery. Honeycomb's flexibility in defining Service Level Indicators (SLIs) and setting SLOs makes it easier for teams to maintain their desired levels of service reliability.

The Game Developers Conference (GDC) is an annual event that serves as a key platform for gaming industry leaders, enthusiasts, and developers to explore trends, network, and celebrate industry achievements without focusing primarily on technology promotion. This year's conference highlighted several key themes, including the importance of fostering safe and diverse gaming communities, the transformative potential of virtual reality in gaming, and the recognition of creativity and technical excellence through the Game Developers Choice Awards. Honeycomb announced its support for the AWS for Games initiative, emphasizing the role of observability tools in enhancing game performance by providing fast feedback loops. Charity Majors, Honeycomb’s CTO, delivered a talk on how understanding player experiences through observability can lead to more productive engineering teams and improved player satisfaction, illustrating the mutual benefits for developers and gamers. The event underscored the GDC's significance as a premier gathering for learning and discussing all aspects of video games.

Honeycomb Pro accounts are now enhanced with access to Honeycomb Metrics and two Service Level Objectives (SLOs), features previously exclusive to Enterprise accounts, though this change accompanies an upcoming price increase for Pro plans starting April 26, 2022. Current Pro customers, as well as those who sign up before the new pricing takes effect, will retain their existing rates until April 2023, allowing them to enjoy the expanded capabilities for a year without additional cost. Honeycomb Metrics enables users to integrate essential system metrics with application observability data for a holistic view of application performance and dependencies, utilizing tools like OpenTelemetry, Prometheus, or AWS Cloudwatch for data input. The SLOs feature helps organizations prioritize and debug business-critical issues by setting performance thresholds and error budgets, thus facilitating collaborative adjustments in engineering priorities. To aid users in maximizing these new offerings, Honeycomb is hosting events, such as discussions with Heroku engineers and a fireside chat with Honeycomb's leadership, focusing on practical applications and strategies for incident response using SLOs.