December 2024 Summaries

Modal has introduced a new inference-focused accelerator, the NVIDIA L40S GPU, priced at $1.95/hr, which offers substantial performance benefits over their current most popular accelerator, the NVIDIA A10 GPU. The L40S provides twice the on-device DDR6 random access memory of the A10, allowing users to run larger models on large inputs without a throughput-killing offload to CPU RAM. This results in a 40% speedup for memory-bound jobs and over a 100% speedup for compute-bound jobs using 16bit Tensor Cores. The L40S also outperforms the A10 in terms of streaming multiprocessor architecture, compute capability, and GPU RAM, with improved bandwidth and arithmetic capabilities. Modal users can now access this new accelerator through their platform, with $30/month in free compute available for sign-up.

This fall, a company went on an offsite to Ericeira, Portugal, where they organized an internal hackathon with around 30 employees. They were impressed by the results and wanted to share some of their favorite projects that used Modal in creative ways. The teams built various applications, including an AI agent app called Browserman, which navigated the internet to complete tasks, a real-time translation app called Glodal, and a workflow orchestrator called Waluigi. Another team developed a pytest plugin that parallelized test suites on Modal, shortening runtime from minutes to seconds. The hackathon showcased the potential of Modal to turbocharge developer productivity across diverse use cases in AI, arts, and software development.

Modal partnered with a top soccer team, AFC Richmond, to help them process and analyze large amounts of tracking data from matches. The current system was not well-suited for this task due to limitations in scalability and cost-effectiveness. Modal's serverless batch processing on GPUs provided a flexible infrastructure that reduced costs by 50% and enabled the team to scale automatically based on data volume. The partnership also led to the development of a lightweight in-memory vector DB, allowing the coaching staff to make queries based on semantic similarity of embeddings generated during analysis. This solution enables AFC Richmond to gain valuable insights into player performance and make informed decisions for future matches.

At Modal, a high-availability VPN proxy called vprox was built using Go and WireGuard. This proxy allows containers to funnel outbound traffic through static IPv4 addresses, ensuring consistent source IP addresses for outgoing internet traffic. The proxy uses SOCKS5 proxies initially but later switched to WireGuard, which provides better security and consistency. To configure networking for the proxy, a policy-based routing system was implemented, allowing traffic from one container to go through a designated VPN interface without affecting its neighbors. This setup is essential for multi-tenant Modal workers that run gVisor sandboxes on shared hosts. The proxy server uses sysctl settings to relax reverse path filtering and ensure reliable operation across Linux distributions. The vprox control plane was open-sourced, allowing developers to configure various aspects of the networking system, including IP discovery and client reconnection.