RespanがClickHouse CloudでLLMオブザーバビリティをスケールさせる方法

Respan, previously known as Keywords AI, manages a massive large language model (LLM) observability framework, handling around 50 million events daily via ClickHouse Cloud. Initially using a Postgres-based setup, they transitioned to ClickHouse Cloud to accommodate the growing scale and to address Postgres's limitations in handling high write loads. This new architecture supports high-throughput ingestion and rapid analytics, utilizing incremental materialized views for efficient data aggregation, which ensures quick dashboard performance despite data expansion to billions of rows. Respan's platform offers visibility into performance, evaluation, and prompt management for LLM applications, routing requests across multiple models and providers. The company, which participated in the YC W24 batch, has rapidly grown to handle approximately 1 billion monthly requests. Their system design strategically separates real-time metrics from user metadata to optimize query performance and uses a partitioned MergeTree table structure for efficient data querying. By avoiding costly consistency processing and enabling asynchronous data pipelines, Respan maintains predictable and memory-efficient trace analysis. The simplicity and scalability of ClickHouse's indexing model reduce operational complexity, allowing Respan to scale with increasing usage without frequent architectural overhauls.