Ensuring reliable OpenTelemetry ingestion at scale

ClickHouse Cloud’s LogHouse observability platform now ingests 50 million OpenTelemetry events per second and stores 177 PiB of uncompressed data, requiring an ingestion architecture that can withstand unpredictable bursts and database backpressure without data loss or excessive permanent capacity. Initial agent-to-gateway collectors relied on in-memory queues and failed during outages, while a local disk-based write-ahead log improved durability but created slow, FIFO backlog recovery, operational complexity from persistent volumes, and delays to fresh telemetry. Rather than operate Kafka or another streaming platform solely for this use case, the team built a stateless OpenTelemetry collector design that routes data directly to ClickHouse under normal conditions and automatically fails over to S3-compatible blob storage when ClickHouse is unhealthy. Blob-storage event notifications are delivered through lightweight queues such as SQS, which separate catch-up collectors use to replay overflow data into ClickHouse after recovery, ensuring live data is not blocked behind historical backlog. A staging outage test showed that traffic switched automatically to S3 and later resumed direct ClickHouse ingestion while catch-up collectors drained stored data independently. Although the approach requires region-specific buckets, queues, notifications, and more complex collector configuration, it has scaled from 10 million to 50 million events per second, reduced operational burden and costs, and is being extended to Clickstack Cloud’s managed OTLP ingestion service.

No tracked trend matches for this post yet.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.