Lambda on hard mode: Inside Modal's web infrastructure

Modal developed modal-http, a Rust-based HTTP and WebSocket service that translates incoming web traffic into its existing serverless function-call system, allowing users to deploy endpoints ranging from simple APIs to long-running, GPU-intensive workloads with large inputs and outputs. Unlike many serverless platforms with strict execution, memory, bandwidth, and payload limits, Modal supports containers with up to 64 CPUs, 336 GB of memory, and eight Nvidia H100 GPUs, requiring an architecture that can efficiently autoscale while streaming large requests and responses. The service represents HTTP interactions as serialized ASGI-style events, enabling request and response streaming, backpressure, error propagation, request bodies up to 4 GiB, and unlimited streamed responses. It handles browser idle timeouts for lengthy work through periodic HTTP redirects and extends the same event-based model to WebSocket handshakes and bidirectional messaging. Modal runs modal-http behind a TCP load balancer, Caddy, and Kubernetes for conventional ingress infrastructure, while its custom serverless runtime handles compute separately; it currently serves ingress from Ashburn, Virginia, with additional regions planned. The company reports that replacing its prior Python-based ingress with this Rust implementation reduced production 502 errors by 99.7% and established shared infrastructure for web functions and remote function calls.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.