November 2022 Summaries

Honeycomb has expanded its integrations with AWS services, allowing for a broader range of AWS data to be sent to Honeycomb using enhanced BubbleUp features for faster identification and debugging of issues. Previously limited to a select number of AWS services, Honeycomb can now receive logs from any service publishing to CloudWatch Logs or Amazon S3, as well as metrics from services publishing to CloudWatch Metrics, utilizing serverless technologies like AWS Kinesis Data Firehose and AWS Lambda to streamline data streaming without additional infrastructure. These integrations facilitate the analysis of structured logs, helping users to quickly visualize and address anomalies in their applications and infrastructure. The setup process for these integrations has been simplified through modular "one-click" setups using AWS CloudFormation or Terraform, enabling efficient data transfer from AWS to Honeycomb. While the primary focus is on debugging application performance issues, users are encouraged to selectively enable data from AWS services critical to their operations to optimize costs and Honeycomb event volume. Honeycomb's ability to transform unstructured AWS logs into structured data supports observability, with ongoing efforts to expand the range of AWS services covered by these integrations. Existing Honeycomb Pro and Enterprise users can fully leverage these new capabilities, while Free users are limited to log integrations, and support is available through the Pollinators Slack group for further assistance.

A new Staff Site Reliability Engineer at Honeycomb shares insights from their experience with the company's incident retrospective process, highlighting its focus on learning rather than immediate remediation. During a retrospective meeting for a recent ingestion delays incident, ground rules were emphasized to foster a blame-aware environment where participants assume everyone's actions were well-intentioned based on available information. This approach avoids counterfactual questions that can hinder investigation by introducing blame, and instead encourages asking questions to improve understanding of the system's actual workings. The engineer notes that reading these ground rules aloud at every meeting is crucial to maintaining a constructive tone and mindset, creating a safe space for learning and collaboration. Honeycomb's culture of continuous learning in incident reviews is appreciated and aligns with best practices such as those in the Etsy Debriefing Facilitation Guide and the Howie guide by Jeli, marking a shift from focusing on action items to fostering understanding and improvement.

The introduction of Environments & Services in Honeycomb has led to an increase in the creation of smaller datasets, posing scaling challenges that were addressed by leveraging Honeycomb's telemetry to optimize data distribution across Kafka partitions. As events in Honeycomb are organized into datasets corresponding to services, load balancing among the Kafka partitions is crucial, and when a dataset's traffic grows, additional partitions are added to manage the load. SRE Fred Hebert developed the "Fractional Partition Weight" measure to identify the best datasets to expand, ensuring efficient traffic distribution while avoiding overloading other partitions. The approach considers the diminishing returns of adding more partitions, emphasizing the need for linear increases to effectively manage traffic. Originally, manual analysis and admin UI adjustments were used to manage dataset partition changes, but with the new Query Data API, Honeycomb improved its tool to automate recommendations based on comprehensive data from their dogfood environment, enhancing reliability and reducing toil. The solution illustrates the complexities of scaling partitioned setups and highlights Honeycomb's commitment to improving observability tools, inviting users to explore their free tier and recent feature enhancements.

For software developers utilizing CI/CD pipelines, efficient build pipeline performance is crucial for productivity and rapid feature deployment. The article introduces a new hands-on guide to optimizing CircleCI build pipelines using the Honeycomb CircleCI Buildevents Orb, a tool that generates distributed traces to help visualize performance and identify bottlenecks. This orb simplifies the integration of third-party tools by condensing configuration processes into manageable snippets, enhancing the build experience. By employing distributed tracing, developers can dissect individual build steps to identify inefficiencies, similar to breaking down tasks into granular components. The guide offers a practical walkthrough for using these tools with a sample app, aimed at both beginners and experienced users, and requires free accounts on Honeycomb and CircleCI. The instructional content not only provides step-by-step instructions but also encourages engagement through various community channels.

October was a busy month for Honeycomb, featuring significant updates and new integrations aimed at enhancing user experience and functionality. Key developments included the launch of the Service Map feature, which provides dynamic architecture diagrams from telemetry data, and a substantial overhaul of the BubbleUp tool for identifying common dimensions in problematic events. Honeycomb also introduced a new integration with ServiceNow for quicker issue diagnosis and a Gradle plugin for automatic OpenTelemetry trace generation. Documentation updates were released to facilitate migration from Honeycomb Classic to the new Environments model, and support for OpenTelemetry logs was introduced. Honeycomb expanded its CI/CD capabilities with buildevents, offering integration guides for CircleCI, GitHub Actions, and GitLab CI/CD, and announced the general availability of its .NET OpenTelemetry distribution. Additionally, user experience improvements were made, including updates to the derived columns editor and options for sending logs or traces. The full list of updates is available in Honeycomb's changelog and related platforms.

The text discusses the author's observations and experiences with Service Level Objectives (SLOs) at Honeycomb, contrasting it with traditional approaches inspired by Google's SRE book. While traditional SLOs are often strict and tied to SLAs, Honeycomb adopts a flexible approach where SLOs serve as signals or alerts rather than rigid commitments. This flexibility allows on-call staff to manage alerts and engage in discussions about service expectations with stakeholders. The focus is on customer-oriented SLOs that measure what customers care about, providing actionable insights and maintaining high internal standards without directly tying them to customer SLAs. Honeycomb's approach emphasizes the importance of context and collaboration across teams to define relevant metrics, ensuring that alerts are meaningful and actionable. The text highlights Honeycomb's internal SLOs, like their end-to-end checks SLO, which continuously tests their core product to ensure data availability and prompt action if delays occur. This method reflects a broader trend in the organization to enhance service reliability while remaining adaptable to changing needs.

Databases that charge by request, such as AWS DynamoDB and Azure CosmosDB, have gained popularity due to their scalability, as costs align with usage, unlike traditional models that rely on provisioned compute infrastructure. These databases require optimized queries to manage costs, as inefficient queries can lead to higher expenses. Tools like Honeycomb can be used for monitoring and analyzing query costs, employing features such as heatmaps and BubbleUp to identify cost patterns and optimize usage. Effective strategies include using Point Reads for predictable costs, reducing the size of indexed documents, and considering caching solutions like CosmosDB cache or DynamoDB's DAX for read-heavy workloads. Understanding and monitoring query performance at a granular level is crucial for cost management, offering insights into optimizing database usage and reducing expenses.