Company
Date Published
Author
Kahn Chen, Matt Mangia, George Shu, Gil Friedlis, Kenway Chen
Word count
1512
Language
English
Hacker News points
22

Summary

Tencent, a leading internet-based platform company, has significantly advanced its use of Apache Kafka to enhance the scalability and fault tolerance of its data infrastructure supporting its various digital products, including WeChat. The Platform and Content Group (PCG) at Tencent has developed a federated Kafka solution that incorporates a proxy layer to manage multiple Kafka clusters, thereby overcoming limitations in scalability, customization, and service level agreements inherent in using a single Kafka cluster. This setup allows Tencent to handle trillions of messages daily across hundreds of clusters, with improvements in real-time analytics and feature engineering. Despite challenges such as message ordering issues and the need for frequent updates to the proxy broker interface, Tencent's approach aligns with ongoing developments in the Kafka community, particularly around Tiered Storage and the removal of ZooKeeper. The company's successful implementation demonstrates both its commitment to pushing Kafka's capabilities and its potential contributions to the broader Kafka ecosystem, as it continues to explore further innovation and integration.