March 2023 Summaries

The Twelve-Factor App methodology, developed by Heroku in 2011, provides a framework for building scalable, independent, and composable applications, emphasizing principles that are independent of specific implementation details. While the methodology has remained relevant over the years, certain aspects, particularly Factor 11 concerning logs, have evolved with technological advancements. Originally, Factor 11 focused on treating logs as event streams and using stdout for log output to ensure consistency across services, but this approach is becoming outdated due to inefficiencies in text conversion and serialization. OpenTelemetry now offers a more modern solution by unifying telemetry signals such as logs, traces, and metrics across different languages and frameworks, providing a consistent format for backend observability platforms via protocols like gRPC and HttpProtoBuf. This evolution suggests that OpenTelemetry could supersede the original logging practices outlined in the Twelve-Factor methodology, encouraging a shift from stdout to more efficient data transmission methods.

Observability in microservices can be effectively initiated by focusing on tracing rather than metrics, with OpenTelemetry recommended as a starting point for implementing SDKs for tracing in applications. By using tools like Jaeger to manage trace data, developers can gain detailed insights into user experiences, which are often lost when solely relying on metrics that focus more on infrastructure aspects like CPU usage. As understanding deepens, additional tracing information can be added to increase visibility, and various advanced observability techniques, such as service level objectives and service maps, can be explored based on the application's scale. For those seeking to expand their knowledge, literature by experts like Charity Majors, George Miranda, and Liz from Honeycomb is suggested, offering insights beyond beginner guides.

Open source projects often face challenges with aligning incentives between developers, contributors, and end users, especially when projects are maintained by unpaid solo developers or involve larger corporate participation. While some attempts, like Tidelift and GitHub Sponsors, aim to bridge this gap, there are still notable discrepancies, particularly in cloud-native projects. Corporate-backed initiatives often see contributions on employer-paid time, with large tech companies and smaller developer-centric businesses benefiting most from cloud-native technologies. However, to broaden adoption and address misaligned incentives, projects like OpenTelemetry have initiated end-user engagement strategies, such as community surveys and monthly discussions, to gather feedback and improve project implementation. These engagements have helped generate valuable insights and incremental improvements, though challenges remain in effectively routing feedback and addressing non-code contributions. Efforts to enhance community involvement and education, through speaker series and demo developments, have shown promise in moving adoption beyond early adopters. Organizations are encouraged to map out stakeholders and align goals to foster inclusive participation and drive project growth, emphasizing lightweight experimentation and identifying under-represented contributors.

In an effort to redefine and update the SRE (Site Reliability Engineering) charter at Honeycomb, the team, now comprising four engineers and a manager, has evolved its mission to better align with their current responsibilities and aspirations. The updated charter emphasizes three key areas: championing reliability and scalability, leading on-call and incident practices, and providing a system-level perspective to the organization. These areas reflect the team's focus on long-term, holistic system views, cross-pollinating silos, and influencing engineering practices toward sustainability and scalability. The charter also includes the secondary objective of generating content such as incident reviews and blog posts to offer value both internally and externally. This new charter is positioned as a living document to adapt to organizational changes, ensuring that the SRE team remains aligned with evolving priorities and high-leverage activities.

Honeycomb has introduced a range of new features and improvements to enhance user experience and data management, including a Service Map for visualizing service interactions and an updated BubbleUp feature to identify outliers in any query. They have refined UI elements for better legibility and functionality, such as clickable chart lines and improved click-to-trace functionality. API key management has been streamlined with creation date displays, and markdown support has been added to SLOs and Triggers for richer alerts. Honeycomb has also expanded its telemetry offerings with an OpenTelemetry distribution SDK for Node.js and introduced environment-derived columns to simplify data configurations across services. On the integration front, new Terraform integrations and AWS support have been rolled out, allowing users to manage Honeycomb configurations programmatically and enhance AWS log and metric integrations. Additionally, Honeycomb Metrics are now available in the free tier, and a new Service Map sandbox offers a hands-on exploration of its features, while a learning app called Happy o11ydays provides a creative way to engage with Honeycomb's query and trace capabilities.

Deploying software is often seen as an inefficient and risky way to manage changes in user-facing code because it conflates the processes of deployment and release, leading to potential issues such as timing mismatches and errors that require complete rollbacks. Deployments are intended for frequent, small updates that are invisible to users, whereas releases involve significant changes to user experience and often require broader coordination. Feature flags offer a solution by allowing code to be deployed continuously while controlling user-visible changes with simple toggles, enabling safer, more flexible releases managed by non-engineering teams. This practice supports progressive delivery and testing in production, allowing for gradual rollouts and targeted testing, ultimately making the software development process faster, safer, and more efficient. Adopting feature flags does not need to be a large undertaking and can start with simple configurations, leading to improved collaboration across teams and an accelerated development cycle.

An individual experiencing the "Missing 'x-honeycomb-dataset' header" error while sending traces to Honeycomb should create a new environment and obtain a new API Key to resolve the issue. Previously, Honeycomb required the x-honeycomb-dataset header to specify where to store data, but since April 2022, this is determined by the service.name field in events, with environments now being part of the framework. Users are advised to establish environments named according to their app's context, such as "prod" or "test," allowing Honeycomb to automatically determine the dataset based on the new API Key. However, when sending metrics, the x-honeycomb-dataset header is still necessary. Legacy and Classic environments can be identified by their lengthy, lowercase hex API Keys, while current ones are shorter with capital letters. For those using gRPC or HTTP, errors such as status code 16 or a 401 Unauthorized may occur if configurations are incorrect.

Professionals are adept at analysis, which is effective for understanding simple or complicated systems where parts can be mastered and controlled. However, complex systems, characterized by learning, change, and dynamic relationships, require the skill of sense-making, a natural human ability to form ideas and understand systems through observation and discussion. Unlike analysis that promises control, sense-making is about forming theories, asking questions, and gaining a good-enough understanding to influence outcomes, as illustrated by examples in team dynamics and software development. In software, readable and testable code supports analysis, while observable code aids sense-making by providing insights through traces and queries. Both analysis and sense-making are crucial for software development, with the former being a learned skill and the latter a natural human trait that can be honed, allowing for better navigation and influence within complex systems.

Debugging systems with the introduction of a message queue can complicate trace propagation, as traditional methods using distributed traces may not fully apply. The post explores the differences between synchronous systems, where user actions map directly to traces, and asynchronous systems with message queues, where mapping becomes more complex due to the indirect correspondence of user actions to events. It discusses the concept of trace propagation and how OpenTelemetry and W3C tracing standards help in linking spans and traces, particularly with message-driven architectures. The article provides examples of different scenarios, such as email sending and account reconciliation, to illustrate the challenges and solutions for tracing in event-driven systems. It highlights the use of span links versus parent-child relationships to maintain context across systems, using Azure Service Bus as a practical example to demonstrate how metadata and activity contexts can be used to propagate trace information. The post emphasizes the importance of designing debugging experiences to accommodate the complexities of asynchronous messaging and suggests using OpenTelemetry propagation APIs for consistency across different applications and languages.

Distributed tracing is a critical tool for monitoring and understanding the flow of requests through distributed systems, significantly enhancing debugging and troubleshooting processes by pinpointing performance-affecting areas without relying solely on traditional logging methods. Instrumentation plays a vital role in this process, allowing for the parsing and analysis of high-cardinality data at granular levels, with OpenTelemetry SDKs recommended for implementation due to their vendor-neutral nature. Companies like Slack, HelloFresh, and Intercom have successfully integrated distributed tracing to improve their operations, with Slack drastically reducing flaky test rates and accelerating CI processes, HelloFresh streamlining tool maintenance and incident resolution, and Intercom enhancing observability workflows and team collaboration through proof-of-concept visualizations. Honeycomb, a platform supporting these implementations, has demonstrated its value by enabling faster problem-solving and efficient observability, encouraging teams to adopt distributed tracing widely for improved system performance and clarity.