How to Tune Kafka Connect Source Connectors to Optimize Throughput

This blog post discusses how to tune your source connectors in Kafka Connect to improve throughput by increasing the number of records pulled from the endpoint and reducing the time it takes to apply transformations. The authors walk through an example using a JDBC Source connector, explaining that the Converter and Transformations stages are not tunable, but the Connector and Producer configurations can be modified to optimize performance. They provide a step-by-step guide on how to determine which configurations to change first, starting with producer-level metrics such as batch-size-avg and records-per-request-avg, and then tuning connector-level metrics like poll.interval.ms to improve throughput. The authors demonstrate an example where the throughput is increased by 57% from 50.9 MB/s to 88.2 MB/s by tuning both the producer and connector configurations. They emphasize that this methodology can be applied to any source connector and that understanding how a connector works is essential for effective tuning.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.