How we use ClickHouse as a real-time stream processing engine

Mux faced challenges with their existing video data processing pipeline as the company scaled, necessitating a rearchitecture to improve data freshness, query performance, and operational efficiency. The old system, reliant on Flink and Postgres, struggled with issues like manual scaling during traffic surges and complex operational management due to Flink's steep learning curve and Java implementation, which was inconsistent with Mux's predominantly Go-based stack. To address these limitations, Mux explored three options, ultimately selecting ClickHouse as both their stream-processing engine and persistent data store, replacing Flink and Postgres. This choice was driven by ClickHouse's ability to handle complex calculations over large datasets and the team's existing expertise with the technology. Despite challenges like translating Java logic to SQL and maintaining a high level of data accuracy, the new system improved data freshness from hours to minutes and enhanced query performance. The transition involved meticulous testing and validation to ensure data integrity, employing a query shadowing system to compare outputs from the old and new systems. The successful implementation not only simplified the architecture but also unlocked new features, such as more granular billing representations and enhanced dashboard visualizations, ultimately allowing Mux to meet customer demands more effectively and efficiently.