September 2024 Summaries

Batched is an open-source, lightweight Python library designed to add dynamic batching to transformer models or other functions without requiring developers to adopt a larger inference ecosystem. Dynamic batching collects incoming inference requests briefly and processes them together, improving GPU utilization and throughput by taking advantage of parallel computation, while typically adding only milliseconds of waiting time. The library wraps existing model methods, such as a SentenceTransformer encoding function, and can be integrated into APIs like FastAPI with minimal code changes. Internally, Batched queues calls until a configurable wait period expires or a maximum batch size is reached, then combines requests and can pad model inputs when needed. Its authors report benchmark results of up to a tenfold throughput improvement, lower overall request latency, and better GPU utilization, though they note that outcomes depend on hardware, models, and traffic patterns and recommend testing in individual workflows.