Building a Resilient System: Our Journey to Observability at Intercom

Kesha Mykhailov, a Senior Product Engineer at Intercom, explores the company's journey in enhancing its culture of observability to improve system resilience and customer experience. Observability at Intercom is defined as a continuous process of asking and answering questions about the production environment. Initially reliant on metrics, Intercom identified the need for more comprehensive observability with high-cardinality attributes and adopted tracing telemetry to provide richer insights. A proof-of-concept using Honeycomb's existing tracing library was implemented, facilitating a smoother observability workflow and enabling engineers to engage with data more effectively. The transition to traces involved an extensive enablement program, involving various stakeholders, to maximize the adoption of new tools. Intercom evaluated potential vendors based on criteria such as exploratory workflows, sampling and retention controls, and pricing, ultimately choosing Honeycomb for its alignment with their needs. Post-implementation, the focus shifted to increasing adoption through initiatives like tracing in development environments and Slackbot query shortcuts. Observability tooling's ROI was measured using engagement metrics, revealing unexpected benefits in cost management and security audits. Intercom plans to continue integrating observability practices into its operations, inviting others to join their journey through a free tier offering.