August 2019 Summaries

Honeycomb's unique on-call rotation system involves every team member, fostering a collaborative culture where engineers learn firsthand about the challenges of operating their code in production. This approach contrasts traditional models where roles are distinct and separate teams handle development and operations. By having a two-week rotation, with a primary and secondary on-call role, Honeycomb ensures that knowledge and responsibilities are shared, and critical issues are addressed swiftly, minimizing stale tickets. This method, supported by a centrally shared on-call playbook and a culture of blamelessness, reduces stress and enhances learning across the team, leading to improved software delivery and operational performance. The company's emphasis on real-time observability and a loosely-coupled architecture aligns with findings from the DORA State of DevOps report, which highlights the benefits of such structures for rapid, independent system updates. Honeycomb's approach demonstrates that, while initially daunting, the collective on-call experience strengthens team dynamics and broadens understanding, ultimately benefiting both the organization and its customers.

Evan Shaw, Lead Engineer at vendhq.com, describes the transition to using GraphQL for Vend's APIs and the subsequent challenges and solutions in monitoring server performance. GraphQL was praised by frontend engineers for its flexibility and structured schema compared to RESTful APIs, but it introduced complexities on the backend, particularly in aggregating meaningful metrics due to its single HTTP endpoint that could handle numerous query types. Initial attempts to monitor performance with Datadog and AWS X-Ray proved inadequate; however, the implementation of Honeycomb tracing, using the Go library gqlgen, provided better insights, though it initially resulted in traces with excessive segments. By refining the tracing approach to only include complex fields, Shaw identified and addressed inefficiencies such as unnecessary queries and ineffective dataloader configurations, significantly improving internal request handling. Despite ongoing challenges with trace focus and retention, the transition has enhanced Vend's ability to understand and resolve server-side issues, encouraging further exploration and optimization of production environments.

Increment magazine includes an article by Charity Majors, CTO and co-founder of Honeycomb, discussing the concept of "testing in production." Majors emphasizes that testing in production can be a powerful tool, but the industry's failure to recognize and properly equip for it poses significant challenges. She argues that acknowledging and investing in the necessary tools and training to conduct safe and effective production testing is crucial for overcoming these obstacles. The article, titled "I test in prod," provides further insights into her perspective on the subject.

The blog post, originally published in August 2019, details a method for using Honeycomb to manage and optimize AWS bills, which are known for their complexity. The authors describe their process of integrating AWS Cost and Usage Reports with Honeycomb, highlighting its benefits like ease of access for engineers and the ability to set up automated alerts for unexpected cost changes. They explain the steps of configuring AWS accounts to generate detailed CSV reports, downloading these reports, and sending them to Honeycomb for analysis. The post emphasizes the importance of using Honeycomb’s query features to track and understand spending patterns, particularly for users with Reserved Instances. The authors caution users spending over $100,000 per month on AWS to consider alternative techniques due to the scale and complexity involved. Additionally, the post discusses setting up triggers for cost changes and suggests tagging infrastructure for more detailed usage breakdowns. The blog is aimed at empowering teams to gain better visibility into their AWS expenses, even offering tips for further customization and optimization.

Charity Majors, in her article for The New Stack, examines the evolution of observability, covering its history, key contributors, and intended purpose, while also offering her perspective on its future trajectory. She predicts that in the coming three years, traditional categories such as Application Performance Monitoring (APM), monitoring/metrics, and logs will likely merge into a singular category known as observability, reflecting a shift in how these tools are utilized and integrated within the tech industry.

Honeycomb's journey to integrate continuous integration/continuous deployment (CI/CD) practices into its infrastructure as code using Terraform highlights significant operational improvements and cost savings. Initially, the company faced challenges with uncoordinated Terraform deployments and config drift, leading to reliability risks and a lack of confidence among engineers. By implementing a centralized deployment system and safety measures such as deletion protection and usage of Sentinel, Honeycomb strengthened its Terraform processes. The adoption of AWS Spot Instances further enhanced cost efficiency by allowing preemptible instances at reduced prices, resulting in a substantial reduction in compute costs. This shift enabled more frequent and less risky deployments, allowing Honeycomb to align infrastructure development with business goals effectively. Additionally, the investment in modern Terraform tools and practices led to successful cost management, notably decreasing the AWS bills, which is pivotal during investor evaluations. The initiative not only streamlined operations but also underlined the importance of proactive financial oversight and infrastructure agility in sustaining the company's growth and operational efficiency.