January 2024 Summaries

Honeycomb has introduced a new data residency option in Europe, allowing customers to store their data either in the US, Europe, or both, in compliance with the EU-US Data Privacy Framework. This expansion enables customers to adhere to regional data privacy regulations while using Honeycomb's observability solutions, with automatic incorporation of a Data Processing Agreement into the Terms of Service. Both European and North American instances offer identical features and pricing, and Honeycomb provides extensive support and guidance to EU customers, including expert advice on OpenTelemetry instrumentation and telemetry pipeline setup. Customers operating in both regions can choose to separate their data to meet compliance regulations, and Honeycomb plans to offer a self-serve option for European data residency in the future.

Honeycomb, a leading observability platform, has announced the deployment of its SaaS cloud service in the European Union, allowing EU companies to adopt observability while complying with GDPR and local data residency regulations. The move supports Honeycomb's commitment to expanding its presence in Europe by appointing Dave Roberts as the Regional Vice President for the EMEA region, who will utilize his extensive experience to enhance customer onboarding and education. The platform, known for its ability to manage complex cloud-native systems and provide real-time insights, empowers engineering teams to optimize performance and drive innovation. By addressing challenges in distributed cloud systems, Honeycomb enables companies to handle large volumes of telemetry data and meet high user expectations, thereby enhancing their operational performance and competitive edge.

The text discusses the rising costs and diminishing returns associated with observability tools in software engineering, highlighting the inefficiencies of Observability 1.0, which relies on separate tools for metrics, logs, and traces. This model results in high costs due to the need to store data in multiple formats and the difficulty in correlating data across tools, leading to increased complexity and cognitive load for engineers. Observability 2.0 is presented as a solution, utilizing a single source of truth with wide structured log events, allowing for better data correlation and more efficient telemetry management. This approach not only reduces costs but also enhances the engineering team's ability to iterate quickly and confidently, improving the overall software development lifecycle. The text suggests that while Observability 2.0 may not be cheap, it offers greater value by aligning costs with the benefits derived from telemetry, thus enabling faster and more informed decision-making in software development.

Effective alerting involves understanding and managing the balance between signal and noise within socio-technical systems to ensure alerts are meaningful and actionable. A well-calibrated alert system should accurately distinguish between valid events and noise, minimizing false alarms and reducing alert fatigue among operators. Various signal categories, such as saturation thresholds, error rates, and performance indicators, must be carefully managed to avoid overlap and ensure relevant alerts. The iterative approach recommended by Honeycomb emphasizes continuously refining alerts through feedback and analysis, considering both human and automated responses. This process involves assessing the significance of alerts, adjusting thresholds, and potentially categorizing observability data to enhance clarity. While the challenge of overlapping signal types is largely unavoidable, the ultimate goal is to ensure that alerts effectively demand attention when necessary and contribute to improved system reliability and performance.

Jeremy Blythe, VP of Engineering at evertz.io, discusses the art of effective code instrumentation using OpenTelemetry, emphasizing the importance of high-quality, full-stack observability for an improved system experience. He highlights the significance of understanding your audience and maintaining consistency in naming conventions across applications, particularly in microservice environments. Blythe underscores the utility of OpenTelemetry's Semantic Conventions for establishing standardized naming across codebases and platforms, while also recommending the use of domain-specific conventions where applicable. He cautions against the pitfalls of inappropriate attribute naming and advises on thoughtful data capture to avoid excessive noise. Blythe also introduces tools like Honey-health and Honey-explore to enhance the observability process by aligning datasets with Semantic Convention models and creating a unified interface for dataset exploration. The article ultimately champions the strategic use of observability to navigate complex systems and improve overall software development practices.

Amidst routine work, a security issue was identified through a pentest, highlighting the challenges of remediating vulnerabilities with limited tooling, particularly concerning Kubernetes pods accessing the Instance Metadata Service (IMDS) in AWS EC2 instances. The focus was on transitioning from IMDS v1, which allows unauthenticated data access, to the more secure IMDS v2, which requires a session token, to prevent unauthorized access by pods. The challenge was tracing the origin of IMDS v1 calls within a Kubernetes cluster without modifying EC2 scripts or relying solely on per-instance data. This led to the development of the Honeycomb Network Agent, an open-source solution that provides enhanced visibility at the network level by capturing packet data and sending it as events for analysis. By utilizing this agent, the team efficiently identified unnecessary IMDS v1 calls, allowing them to block or address them appropriately. Upon ensuring all necessary calls were addressed, the team modified their autoscaling group launch templates to mandate IMDS v2, resolving the security issue and showcasing a blank graph in follow-up queries, indicating successful remediation.

The OpenTelemetry Collector is a crucial component for telemetry pipelines, necessitating heightened security, which can be best achieved by building a custom Collector executable rather than relying on default images provided by the OpenTelemetry team. These default images, including both core and contrib versions, often include unnecessary components that increase the attack surface, posing potential security risks. To address this, a new approach has been developed using a tool that allows users to create a custom Collector by specifying desired components via a manifest.yaml file, thus reducing unused components and improving security. This process can be simplified further with a two-stage build method that facilitates the automatic generation of the manifest file, eliminating the need for deep Go knowledge and manual configuration. By leveraging the "OpenTelemetry Collector Builder Config Builder," users can create a tightly coupled Collector executable in a secure container, making it more accessible for developers and integrable into existing pipelines without compromising security.

For over a decade, application performance monitoring (APM) tools have been popular for tracking app metrics, offering low-effort integration and instant metrics collection, but often leading to unexpected high costs due to inefficient practices in modern architectures. The rise of service complexity and the need for dashboards and custom metrics can inflate costs, forcing teams to choose between visibility and affordability, which can impact product stability and user trust. Modern observability solutions like Honeycomb address these challenges by eliminating the need to store data in multiple formats and focusing on event-based pricing, using wide, attribute-rich events that integrate log and trace data for real-time analysis without pre-storing metrics. This approach, supported by scalable tools like OpenTelemetry, allows teams to shift from traditional APM practices, leading to significant cost savings and reinvestment into product enhancements. Companies like CCP Games, Intercom, and Slack have benefited from improved instrumentation and problem-solving efficiency, with a notable example being a compliance tech provider that reduced observability costs from 5% to less than 1% of total revenue by adopting these modern practices, thus aligning observability spend with application value while enhancing user experience.

A product manager at Honeycomb discusses the critical issue of burnout within product management teams, emphasizing the need for PMs to actively manage stress and workload to prevent it. The author argues that while product managers hold significant power in setting team dynamics, they often overlook the impact of stress from ambitious timelines and evolving requirements, leading to burnout. Drawing from personal experience and emphasizing a proactive approach, the author has implemented strategies such as making burnout avoidance an explicit team goal, committing to revisiting project requirements and timelines, and prioritizing team members' well-being over product demands. By fostering open communication and allowing team members to consent to increased stress, the author aims to create a healthier work environment. The piece concludes with a call for PMs to recognize their role in shaping work conditions and to consider the long-term effects of sustained stress on their teams.