June 2025 Summaries

Quora uses Modal Sandboxes to power secure execution of LLM-generated code within Poe, its AI chatbot platform, allowing users to run code directly in conversations without exposing Quora’s infrastructure or other users’ sessions. After launching Poe in 2023 and expanding it to millions of active users, Quora needed a solution that combined strong isolation, low latency, and reliability at scale, requirements that would have taken months and substantial ongoing effort to build internally. Modal provided a production-ready system built on gVisor-based container isolation, scalable sandbox creation that Quora tested at up to 1,000 sandboxes per second, and networking controls including tunnels and IP allowlisting. Quora’s familiarity with Modal from its use as Poe’s recommended bot deployment platform helped support the decision, and the company estimates that outsourcing sandbox infrastructure saves the ongoing work of approximately two engineers while enabling future Poe products to use the same technology.

Open-weight generative models and open-source tooling increasingly allow developers to self-host image, music, and text generation APIs, but competitive performance requires specialized inference optimization. Using Black Forest Labs’ FLUX.1-dev image model on Modal, the authors reduced average 1024×1024 image generation latency from roughly 6.75 seconds to under three seconds through a combination of PyTorch compilation, fused query-key-value attention projections, and channels-last memory layouts, which together delivered a 1.5× speedup. They then applied First Block Caching, an approximate technique that skips diffusion steps predicted to have minimal effect on the output, producing a further 2× speedup while accepting limited quality tradeoffs. Because compiler autotuning substantially increased startup times, they also used compiler artifact caches and Modal Memory Snapshots to reduce cold-start latency by about 30×. The resulting autoscaling service is presented as capable of matching proprietary image-generation APIs in speed while potentially offering greater control and competitive pricing.

Modal has launched version 1.0 of its Python client SDK, marking a commitment to greater API maturity, stability, and fewer future breaking changes for developers accessing its managed serverless cloud platform. The release emphasizes predictable, explicit behavior by replacing automatic inclusion of locally imported packages with explicit image configuration, making deployments easier to understand and debug. It also restructures configuration by separating core function settings from specialized features such as concurrency through additional decorators, reducing complexity in the main function decorator. Modal is consolidating container filesystem setup under the Image class, replacing overlapping approaches such as function-level mounts with a single canonical method for adding local files and directories. The company will batch client updates into a more predictable release cycle with documented changelogs, while future work includes SDKs for other programming languages and adapting the platform’s developer experience for AI-driven and agent-based workflows.