May 2024 Summaries

Hunch, a no-code spatial AI canvas supporting models such as GPT-4o, Claude 3 Opus, and Stable Diffusion 3, needed a secure way to execute AI-generated code that could automate tasks, access data, and install dependencies without exposing its platform to unsafe or unreviewed scripts. After finding other sandboxing options too limited, insecure, or difficult to integrate, the company adopted Modal Sandboxes, which provide isolated, ephemeral containers with configurable compute resources, optional network restrictions, dynamic package installation, storage volumes, fast startup times, and automatic scaling. Modal enabled Hunch to add code execution through a small amount of Python integration while avoiding the complexity of building infrastructure itself. Users can now generate scripts for tasks such as sending Slack webhook results, retrieving API or scraped data, and coordinating AI-assisted test-driven development and code review, expanding the platform’s ability to automate complex workflows.

The authors of the blog post fine-tuned a Stable Diffusion model on the Heroicons library to create a generative icon model. They used Modal, a scalable serverless cloud computing platform, to run the fine-tuning training script and deploy the fine-tuned model as an interactive web app. The authors explored different fine-tuning techniques, including full fine-tuning, sequential adapter fine-tuning, and parallel adapter fine-tuning. They found that full fine-tuning worked best for their use-case but also discussed the advantages of parallel adapter fine-tuning methods like LoRA. The authors prepared the dataset by downloading the Heroicons from GitHub, converting them to PNGs, adding white backgrounds, generating captions, and uploading the dataset to the HuggingFace Hub. They then set up a Modal account, created a `TrainConfig` class to hold training hyperparameters, and defined an `AppConfig` class to store inference hyperparameters. The authors fine-tuned the Stable Diffusion model on the Heroicons library using the `train_text_to_image.py` script and saved the fine-tuned model in a Modal Volume. They then mounted the volume to a new Modal `inference` function, which they used to generate icons based on user input. Finally, the authors set up a Gradio UI that called the `Model.inference` function and deployed the app on Modal using one command. The fine-tuned model was evaluated for its performance as an infinite icon library, and the authors discussed the challenges of fine-tuning and the potential of grid searches to scale up the process.

Substack is a popular platform for writers to publish newsletters, employing machine learning (ML) for various purposes such as spam detection and sentiment analysis. The company has moved its ML training and deployment from AWS SageMaker to Modal, a shift that has significantly improved their developer experience, collaboration, and deployment speed. Previously, Substack's pipelines were built on SageMaker and Airflow, but this process was slow and painful due to issues with the developer experience, collaboration, and container startup times. By migrating to Modal, Substack can now iterate quickly, share components across projects, and deploy ML workflows with greater speed and flexibility than ever before.

Modal, a serverless compute platform, now allows developers to select the cloud region for their Functions, making it easier to manage infrastructure. This feature is available on the Team plan and Enterprise plan, and requires an email request to [email protected]. To specify a region, developers can use a line of code, with options ranging from broad regions like "us" or "eu" to specific underlying regions like "us-east-1". Specifying regions can help meet regulatory requirements, reduce egress fees, and minimize latency for applications that require real-time inference. By selecting the desired cloud region, developers can ensure their Functions run in a location that meets their needs, such as running near users or external databases.

Modal’s latest product updates introduce writable AWS S3 and Cloudflare R2 bucket mounts, a new `@web_server` decorator for hosting HTTP servers outside ASGI and WSGI protocols, Okta single sign-on for enterprise users, and a dashboard-based volume file-system viewer. Client enhancements include multi-file app composition, iterable Dict keys, values, and items, isolated input cancellation for functions, hard memory limits, argument support for `Image.run_function`, multipartition queues, sandbox stdin writing, new Dict CLI controls, remote breakpoint support, and larger volume file transfers. The update also highlights recent technical posts covering high-throughput Llama 3 inference, embedding-model fine-tuning, ETL migration, parallel Blender rendering, and Gemma deployment with vLLM, while inviting users to share projects and feedback through Modal’s Slack community.