Acceldata’s Kafka Observability Utility for Topic Lineage - Kapxy

The Acceldata Engineering team developed a utility called Kapxy to identify Kafka Producer-Topic-Consumer relationship metrics. They discovered that the required information was not available in APIs or JMX metrics, and after deep diving into Kafka's internal implementation documentation and network protocols, they found that each Produce request had a "client_id" field (Producer ID) and a "[topic_data]" field containing Topic names. By extracting these metrics from the network packets sent to Kafka broker using their newly developed Kapxy utility and JMX metrics already collected, they were able to plot a chart showing the relationship between Kafka Producer-Topic-Consumer components in Acceldata Pulse's Kafka dashboard.