Model Batch Inference in Ray: Actors, ActorPool, and Datasets

Batch inference in Ray can be implemented using low-level primitives such as tasks, actors, or actor pools, or high-level APIs like BatchPredictor. The choice depends on the desired level of control and complexity. With low-level primitives, control is given over how to execute batch inference, but requires understanding of Ray's core primitives and implementation details. In contrast, BatchPredictor provides a more declarative and expressive API for batch inference, offering automatic scaling and less code. For data scientists and machine learning practitioners who prioritize scalability and ease of use, BatchPredictor is a desirable option.