How Polymarket scaled their data stack with Postgres and ClickHouse

Polymarket, a prediction market platform, faced challenges in handling growing data volumes and computationally expensive analytical workloads with PostgreSQL, which led to timeouts and resource contention. To address these issues, the team, including Senior Data Engineer Max "Primo" Mershon, decided to complement PostgreSQL with ClickHouse, a faster and more scalable data warehouse solution that could manage both internal analytics and user-facing features more effectively. The transition involved integrating on-chain data from Goldsky, web analytics, and off-chain metadata into ClickHouse, allowing for efficient query execution and the creation of dynamic, granular leaderboards. This migration improved performance by reducing load on PostgreSQL and enabling Polymarket to scale their platform, supporting features like categorical leaderboards and complex internal analyses without impacting transactional workloads. The successful implementation of ClickHouse allowed Polymarket to enhance both internal and external functionalities and prepared them for future growth and increased data demands, particularly as they expanded their user base and prepared for a U.S. launch.