Inkwell: Why Your Inference Platform Matters As Much As Your Model

Inkwell, a web application designed to create interactive storybooks in real-time, utilizes Modular Cloud's inference platform to achieve low-latency performance, crucial for generating content on-demand without delays. By tapping into Modular's Gemma 4 31B and Flux2 Dev 32B endpoints, Inkwell efficiently streams story text and images, ensuring a seamless user experience where text appears character-by-character and illustrations materialize as users read. The platform's capability to start image diffusion before text generation completes, enhanced by a 420 ms time to first token (TTFT), enables overlapping processes that reduce perceived wait times. Key optimizations include prefetching potential story paths and caching to minimize delays, with an 85% cache hit rate for user choices. Inkwell's architecture benefits from Modular's efficient API, small runtime footprint, and upcoming server-side intermediate image streaming, which provides a more engaging experience by showing image progression in real-time. Ultimately, the platform's fast response times and support for intermediate states redefine the focus from the model itself to the inference platform's capabilities.