Scaling our Observability platform beyond 100 Petabytes by embracing wide events and replacing OTel

Over the past year, LogHouse, an internal logging platform initially designed to monitor ClickHouse Cloud, has undergone substantial growth and transformation, handling over 100 petabytes of data across nearly 500 trillion rows. This expansion necessitated significant architectural changes and the development of new tools, such as the System Tables Exporter (SysEx), to address the inefficiencies of OpenTelemetry (OTel) in managing high-throughput, high-fidelity system logs. SysEx has enabled a dramatic increase in event processing efficiency, achieving a 20-fold surge in data handling with only a tenth of the previous CPU resource usage. The shift from a one-size-fits-all approach to specialized tooling, combined with the integration of HyperDX, a ClickHouse-native UI, has not only improved data management but also fostered a cultural shift towards high-cardinality, wide-event-based observability. HyperDX facilitates seamless log exploration and correlation, supporting both standardized OTel formats and specialized data from SysEx, thus providing a unified user experience. As LogHouse continues to evolve, future enhancements, including zero-impact scraping and potential migration to JSON, are anticipated to further refine the platform's capabilities and efficiency.