Reducing Kafka connections by 10x with a sidecar pattern

Kafka clients face challenges with excessive TCP connections due to multiprocessing, leading to increased broker memory usage, CPU spikes, and potential data loss. Traditional solutions like adding brokers or increasing broker size proved inadequate due to Kafka's stateful protocol and JVM heap limitations. Instead, implementing a gRPC sidecar per pod streamlined the connection process by consolidating multiple producers into a single producer per pod, significantly reducing connection count, maintaining partition-level ordering, and scaling linearly without refactoring existing services. This sidecar approach resulted in a tenfold reduction in peak connections and a 70% decrease in broker heap usage, eliminating CPU spikes. The sidecar solution required minimal configuration changes, provided operational simplicity, and inspired by Robinhood's Kafka proxy sidecar, successfully improved system performance and reliability.