How the D. E. Shaw group powers high-cardinality observability at scale with ClickHouse

The D. E. Shaw group, a global investment and technology development firm, employs ClickHouse to enhance high-cardinality observability across millions of compute workloads on its internal grid, significantly outperforming alternatives by a factor of seven in evaluations. This transition allowed the firm to handle over 500,000 records per second in ingestion and support long-term capacity planning and strategic decision-making. By maintaining a unique task ID for each workload, they capture telemetry data crucial for understanding compute usage at a granular level, although this approach introduces high cardinality that their existing platform couldn't manage. The implementation of ClickHouse provides improved compression and query performance, enabling the firm to expand observability into distributed tracing and other event-data pipelines. The shift involved translating Prometheus-style queries into SQL, presenting initial challenges but ultimately proving advantageous. ClickHouse's performance, particularly in data ingestion and query efficiency, supports detailed analysis and capacity planning, assisting the firm in making informed decisions regarding compute resource allocation. As the firm continues to explore broader event-data pipelines and structured logs, ClickHouse's role as a unified observability platform grows, demonstrating its capability to handle high-volume, high-cardinality data with the right design and analytical foundation.