April 2021 Summaries

Honeycomb has made its Management API generally available, allowing users to programmatically manage queries, datasets, and other configurations via endpoints, facilitating configuration as code through direct API access or third-party tools like Terraform. This release, which includes CRUD operations for various Honeycomb objects and improvements to existing endpoints, enhances automation possibilities, enabling users to automate common tasks, such as saving queries as boards or triggering alerts for automatic query creation. The API supports Observability-Driven Development by allowing teams to bundle instrumentation with new features, aiding in real-time performance observation during deployments. Customers like Blackbaud are utilizing these new capabilities to enhance their continuous delivery systems, enabling faster and more confident releases. Honeycomb encourages users to explore the API and provide feedback through their Pollinators Slack group.

Honeycomb's platform team recently underwent a challenging migration to a new ingest pipeline architecture for customer events, which highlighted the dynamic changes in the performance envelope of critical systems and their operational impacts. The migration process involved updating the Kafka cluster to utilize tiered storage in Confluent Platform 6.0, allowing for significant reductions in storage costs and operational resources by offloading cold Kafka segments to S3. Despite careful planning, the transition faced several setbacks, including missteps in instance type selection, network resource saturation, and unexpected outages, which underscored the importance of planning for operational concerns beyond steady-state scenarios. The experience revealed the need for manual intervention capabilities when automation fails, as well as the necessity of allocating sufficient resources and gradually adjusting operational limits to manage unforeseen pressures. Honeycomb learned valuable lessons about managing performance and operational envelopes, which they are eager to share with the community to enhance collective understanding of best practices in system migrations.

Honeycomb is organizing a two-day virtual conference called o11ycon+hnycon, scheduled for June 9-10, with an open call for presentations until April 30. The event consists of two distinct conferences: o11ycon, which is vendor-neutral and focuses on observability practices, and hnycon, which centers on Honeycomb's products and customer experiences. o11ycon aims to foster community discussions through interactive formats like open-space sessions and panels, while hnycon features keynote-style presentations and breakout sessions. Both events encourage submissions on themes such as the intersection of observability with other engineering practices, challenges, and solutions in observability, and Honeycomb use cases. Participants can choose the most relevant content and have different platforms to share their stories, with presentation formats tailored to various levels of experience and engagement.

Honeycomb's adoption of Graviton2 has resulted in significant improvements in performance and cost-efficiency, with 92% of their vCPUs now running on the arm64 architecture. This transition, which required minimal codebase changes and primarily involved updating their CI pipeline, has allowed Honeycomb to reduce their ingest workload costs by over 40% compared to previous setups. The company also enhanced their Amazon Machine Image and Chef recipes and developed arm64-compatible agents like honeytail and honeyvent, benefiting the broader community by ensuring tools like osquery and the OpenTelemetry Collector are tested on arm64. The move to Graviton2 facilitated a smoother transition to M1 Apple Silicon Macs for developers and improved performance per thread for their query engine without doubling costs. Their experience has demonstrated that Graviton2 is effective for compute-bound workloads, and Honeycomb supports its integration for enterprise users. Honeycomb plans to further showcase how Graviton2 has enabled them to manage increased traffic while maintaining reliability, latency, and cost-efficiency.