Netflixがペタバイト規模のロギングシステムをClickHouseでどう最適化したか

Netflix processes an astounding 5 petabytes of log data daily, managing an average of 10.6 million events per second using ClickHouse, which is enhanced through three key optimizations. These include faster fingerprinting of logs using a lexer approach, improved serialization with native protocols and LZ4 compression, and efficient querying by sharding tag maps, reducing query times significantly. This architecture allows logs to be searchable within 20 seconds of generation, enabling real-time debugging. Netflix's logging system combines ClickHouse for hot data and Apache Iceberg for cost-effective long-term storage, ensuring engineers can perform interactive debugging and maintain seamless service for over 300 million subscribers worldwide. The success of these optimizations, which transformed Netflix’s logging system into one of the largest and fastest ClickHouse deployments, is attributed to disciplined engineering rather than clever tricks, emphasizing simplicity to handle the company's massive scale efficiently.