September 2022 Summaries

The text discusses the evolving landscape of operations within software engineering, emphasizing the shift towards "platform engineering" as a key development in recent years. This role focuses on minimizing infrastructure management and empowering software engineers to own and manage their code in production, moving away from traditional, siloed operations teams. The concept of platform engineering has emerged to support this transition, facilitating self-service and efficient management of code by leveraging third-party platforms and cloud-native solutions. The narrative highlights the fading relevance of traditional DevOps in favor of a more integrated approach where all engineers are responsible for their code, supported by platform teams that focus on providing a seamless developer experience. The text underscores the importance of operational excellence in increasingly complex systems, while advocating for an efficient distribution of labor and responsibility among engineering teams.

OpenTelemetry, often referred to as OTel, is gaining traction as a unified standard for collecting and sending telemetry data to various backend observability tools, thus allowing development teams to avoid vendor lock-in and gain deeper insights into complex systems. Companies like Jimdo, Upgrade, and Campspot have shared their adoption stories, highlighting the benefits and challenges of implementing OpenTelemetry. Jimdo took a "start small" approach by initially integrating OpenTelemetry with one team, which allowed them to test and compare different observability vendors. Upgrade faced slow tracing yields with their existing setup and used OpenTelemetry to enhance their system observability, while Campspot moved to OpenTelemetry to escape vendor lock-in with their previous APM solution. OpenTelemetry offers SDKs for multiple languages, automatic and manual instrumentation options, and the OpenTelemetry Collector, a versatile tool for processing and exporting telemetry data. These stories underscore the potential of OpenTelemetry to provide comprehensive observability, with Honeycomb being a favored tool among the panelists for achieving effective monitoring and analysis.

In complex, distributed software systems, the effectiveness of monitoring and understanding these systems is enhanced by treating emitted information signals as structured events rather than relying on traditional methods like pre-aggregated metrics or unstructured logs. This approach, advocated by Honeycomb, emphasizes the importance of maintaining the relationships between signal attributes, which constitutes the "information density" essential for comprehending the unique circumstances under which each signal is produced. By preserving these relationships, observability tools allow developers and operators to investigate and understand the contextual activity of software systems more accurately. This nuanced understanding is crucial for maintaining and supporting such systems, as it provides a more comprehensive view of their behavior and the socio-technical environment in which they operate.

Andrew Kuipers from iQmetrix seeks guidance on implementing Availability Checks using Honeycomb for their services, prompting a discussion on the concept and relevance of such checks in modern cloud environments. Availability Checks traditionally involve third-party services pinging endpoints to ensure communication, primarily serving as a last-resort alert system if other monitoring fails. However, the text argues that these checks provide limited actionable insights and suggests that a more effective approach involves monitoring specific metrics and Service Level Objectives (SLOs) tailored to the infrastructure and user experience. By focusing on detailed metrics, such as Azure Front Door and AppService health, and incorporating SLOs for latency and errors, organizations can better pinpoint issues and enhance the reliability of their systems, ultimately providing on-call engineers with more useful information and reducing burnout. The text concludes by emphasizing the importance of a holistic observability strategy that goes beyond reactive measures to proactively improve system performance and user experience.

Hosting the OpenTelemetry Collector in Azure Container Apps offers a streamlined approach to managing containerized applications without dealing with infrastructure complexities. Azure Container Apps provide a managed container runtime environment similar to a Kubernetes cluster, with benefits such as built-in authentication and SSL termination. The OpenTelemetry Collector serves as an aggregation point for system traces, metrics, and logs, and the blog outlines the deployment process using Azure CLI to achieve persistent storage for the Collector's configuration through Azure Files. The steps involve creating a storage account, uploading a configuration file, setting up a container app environment with a storage mount, and deploying the Collector with necessary volume configurations. The post emphasizes the ease of customization and suggests using tools like the yq CLI for efficient YAML file management. It concludes by highlighting the advantages of using Azure Container Apps for centralizing configuration management and hints at future discussions on securing and scaling within virtual networks.

In a detailed incident review, a company faced significant challenges related to query performance and alerting due to complex interactions between hot and cold data storage and the unexpected burden on AWS Lambda capacity. Initially, inaccurate timestamps in a customer's telemetry data led to trigger queries unnecessarily accessing cold storage, tying performance to Lambda usage. An assumption that future-stamps in triggers caused the issue misled the investigation until a fresh perspective identified that repeated backfilling of a single Service Level Objective (SLO) was the true culprit. The incident highlighted the difficulty in managing complex systems where valid but unexpected use cases can lead to resource exhaustion without any technical bugs. The resolution involved correcting the SLO, implementing stricter controls on data handling, and enhancing communication and support for incident management. This experience underscored the importance of diverse perspectives in troubleshooting and the need for adaptable controls in system design to manage unforeseen usage patterns effectively.

In this guide by Andrew Puch and Brian Langbecker, the process of using Honeytail to parse NGINX logs and send this data to Honeycomb for analysis is explained. By leveraging tools such as Node.js, Docker, and Honeytail, users can set up an environment that includes an NGINX proxy alongside a NodeJS Express application utilizing Redis and MySQL. Honeytail is the key utility that enables the tailing and parsing of log files from NGINX, which are then sent to Honeycomb for visualization and analysis. Through detailed instructions, users are guided on how to configure environment variables in a Dockerfile, build a Docker container with Honeytail, and ultimately set up a test application to generate traffic data. The guide emphasizes the importance of specifying the correct log format for parsing and demonstrates using Honeycomb's UI to execute queries and analyze the data, including identifying errors through heatmaps and other visual tools. With this setup, users are encouraged to explore data further and create customized dashboards, taking advantage of Honeycomb's capabilities to gain insights into application traffic.

Honeycomb has announced significant updates to its integration with PagerDuty, enhancing the information provided in PagerDuty notifications and allowing for greater configurability. These updates, available to all users leveraging Honeycomb Triggers and Burn Alerts, include the adoption of PagerDuty's V2 API, which offers new functionalities like customizable alert payloads and configurable severity levels, enabling users to define notifications as Critical, Error, Warning, or Info. The integration now supports the management of email and Slack webhook recipients through a new Recipients API, allowing for precise matching with native PagerDuty IDs. Additionally, Honeycomb graphs can now be included in PagerDuty notifications, offering valuable context directly within alerts. Configuration as code is supported for all integration settings, accessible via the Honeycomb UI, API, or Terraform Provider. These enhancements aim to give engineering teams more observability data, reducing incident resolution times and improving digital service performance, as emphasized by Steve Gross, Sr. Director at PagerDuty. The updated functionality is available for immediate use, with free trials offered for new users.

CI/CD pipelines are crucial for efficient software development, but they can face issues such as slow builds, flaky tests, and integration bottlenecks that hinder productivity and waste developer time. Observability, as demonstrated by Slack, offers a solution by providing insights into these problems. By instrumenting their pipeline, Slack was able to reduce their flake rate from 15% to 0.5%, significantly improving developer efficiency and reducing IT costs associated with re-running tests. Utilizing tools like Honeycomb, Slack was able to trace performance issues and optimize their CI/CD workflows, leading to improved development productivity and a better understanding of pipeline dynamics. This approach not only enhances operational efficiency but also provides valuable insights into system performance, thereby reducing incidents and improving overall software delivery processes.

Honeycomb has introduced several new features aimed at enhancing user experience and data accessibility, including improved filtering capabilities for both the BubbleUp and Raw Data views, which allow users to quickly locate specific information within extensive datasets. An experimental derived column editor has also been launched, offering a more user-friendly interface with features like multiline editing, syntax highlighting, and autocomplete, thereby simplifying the process of creating complex data columns. Additionally, a new copy button in the Honeycomb documentation makes it easier to copy and paste code snippets. These updates are part of Honeycomb's ongoing efforts to keep users informed about product developments, with further details available in their changelog and communication channels like Slack and Twitter.

During a Twitter Space discussion, experts from Honeycomb addressed how to effectively communicate the value of observability to C-level executives by highlighting its impact on business outcomes and system performance. Observability, characterized by high cardinality and dimensionality, allows organizations to gain insights into user experiences and system inefficiencies that traditional monitoring tools cannot provide. Success stories from clients like Intercom and Vanguard demonstrate how observability enabled them to identify and resolve performance issues, ultimately leading to improved customer experiences and cost savings. Executives are particularly interested in metrics such as lead time for changes, deployment frequency, change failure rate, and mean time to recovery, which are positively influenced by observability. As businesses undergo cloud migrations and seek to enhance development velocity, observability becomes crucial for tracking and improving these key performance indicators. Emphasizing outcomes, such as improved customer satisfaction and increased development efficiency, helps bridge the conversation between technical teams and business leaders.