Tuning Apache Kafka Consumers to maximize throughput and reduce costs

New Relic operates a large streaming platform using Apache Kafka, processing petabytes of observability data daily. Key to optimizing Kafka's performance is the effective tuning of Kafka Consumers, which receive data from the stream. The article emphasizes the importance of high-quality instrumentation to monitor and adjust Kafka Consumer metrics for better efficiency and cost management. It provides insights into configuring fetch size and latency settings such as fetch.max.bytes, max.partition.fetch.bytes, fetch.min.bytes, and fetch.max.wait.ms to avoid inefficiencies. The New Relic Kafka UI aids in visualizing these metrics, enabling users to identify and address issues like hitting fetch size limits or inefficient data batching. While the article focuses on the Apache Kafka Java Client, the principles are applicable to other language clients. It also mentions potential future developments for automated tuning solutions and encourages using alerts to detect resource wastage.