How LY Corporation uses ClickHouse to observe one of the largest Kafka deployments on earth

LY Corporation utilizes ClickHouse to manage and observe its vast Kafka deployment, which processes over 1 trillion messages and 2.6 petabytes of data daily. With 24 servers, LY processes 7 million rows per second of API logs for real-time debugging, helping engineers address unique challenges that arise from operating at such a large scale. Born from the merger of LINE and Yahoo! Japan, LY operates a wide array of digital services, including Japan's most popular messaging app and a significant news portal, all interconnected by a massive Kafka platform. This system handles 31 million messages per second, requiring advanced observability to maintain performance and resolve unprecedented issues. LY's observability stack, resembling a research lab, leverages ClickHouse for its SQL compatibility, compression capabilities, and performance, enabling efficient data management and analysis. Engineers use ClickHouse's queryability to diagnose and solve complex Kafka bugs, like a race condition affecting message offsets. ClickHouse's integration into LY’s observability platform provides crucial real-time forensics, allowing the team to maintain one of the world’s largest Kafka systems effectively.