Tuning Apache Kafka Consumers to maximize throughput and reduce costs

New Relic runs a large streaming platform that processes several petabytes of observability data daily, relying heavily on Apache Kafka for efficient data processing and low latency. To maximize efficiency and reliability, it's crucial to tune the Consumers, which receive data from the stream processing applications. High-quality instrumentation is necessary to diagnose performance and resource usage, and New Relic provides a curated UI to collect and analyze metrics. Understanding how to interpret these metrics is key to detecting and remedying common issues such as leaving data on the table or not waiting for more data. By tuning settings like fetch.max.bytes and fetch.min.bytes, and considering factors like compression rate and memory use, developers can optimize their Kafka Consumers for better performance and cost savings. Additionally, New Relic is exploring ways to provide dynamic and automatic tuning for Kafka Consumers with simpler constraints, while also offering alerting tools to help identify wasting cluster or compute resources.