Using ClickHouse to scale your events engine

Lago, an open-source usage-based billing platform, faced performance issues with its Postgres-only database stack as it began processing millions of events per minute. To address this, the company adopted a hybrid database strategy, incorporating ClickHouse, an OLAP database, to handle streamed events efficiently while retaining Postgres for other data needs. ClickHouse's features, such as dynamic materialized views, specialized engines, and vectorized query execution, made it an ideal choice for managing complex analytical queries and large write operations without affecting app performance. The decision to use ClickHouse was influenced by its mature product, community support, and hardware optimizations, although newer solutions like Hydra might have been considered if available earlier. The implementation involves using ClickHouse for ingesting billable events through Apache Kafka, with data processed into a MergeTree table for efficient metric aggregation. The deployment was managed using Altinity Operator in a Kubernetes cluster, balancing cost efficiency and maintenance ease. This shift mirrors a broader trend among open-source projects like PostHog and GitLab, which have also transitioned to ClickHouse to better handle scaling challenges in data processing.