Is Kafka the Key? The Evolution of Highlight's Ingest

Highlight faced significant challenges in adapting to increasing customer data loads due to their initial data ingestion method, which involved streaming data directly into PostgreSQL. This approach caused issues during database migrations, leading to service interruptions. To address these challenges, Highlight explored solutions for buffering data, ultimately opting for a producer-consumer model using Apache Kafka, which allows for large message handling and ordered processing within partitions. Kafka's architecture enabled Highlight to decouple data ingestion from processing, ensuring consistent data flow and scalability. They configured Kafka for optimal performance, focusing on message ordering, data replication, compression, and efficient rebalancing of consumer groups. These enhancements allowed Highlight to scale their operations and maintain data integrity across thousands of messages per second, ensuring reliable service for their growing customer base.