Challenges with Implementing SLOs

Honeycomb's Service Level Objective (SLO) feature was designed to enhance service reliability by providing a mechanism to measure and monitor service quality. The feature evolved with insights from Liz Fong-Jones, an experienced Google SRE, and leveraged Honeycomb's ability to store rich data, enabling unique SLO capabilities. The development process revealed several challenges, such as the need for an intuitive monitoring experience over the creation process and the importance of an alerting system to warn users of potential SLO failures. Early user feedback highlighted the feature's ability to identify key issues, although initial enthusiasm waned due to the absence of alerts. Honeycomb's alert system was refined after a costly AWS incident, emphasizing the balance between accuracy and system efficiency. The experience underscored the significance of volume in SLOs, the importance of testing pathways instead of individual users, and the need for continuous iteration to refine SLOs. Honeycomb's journey with SLOs serves as a guide for others in implementing these observability features, demonstrating the complexities and insights involved in rolling out such a system.