November 2024 Summaries

Mochi 1, an open state-of-the-art video generation model, was recently released by Genmo. The model can generate short video clips from text prompts and is designed to be fine-tuned for specific use cases with the help of LoRA (Low-Rank Adaptation), a technique that reduces memory requirements. By releasing scripts and sample code for fine-tuning Mochi 1 with LoRA, Genmo has made it easier for others to build on top of this foundation model. Fine-tuning allows for superior control of style and increased character consistency across generations, making it useful for specialized video content or proprietary data that defines a brand's style. The process can be resource-intensive, but LoRA reduces memory requirements, enabling fine-tuning on a single GPU with minimal data. Mochi 1 is designed to work seamlessly with Modal, a high-performance AI infrastructure platform that offers scalable resources and enterprise-grade security.

Modal announced a Strategic Collaboration Agreement with AWS to help startups and enterprises build and deploy AI products using its serverless, GPU-accelerated compute platform. Modal enables teams to launch GPU-enabled containers in as little as one second, automate scaling, and avoid managing underlying infrastructure, while AWS supplies security, resilience, and scalable cloud services. The collaboration is intended to strengthen Modal’s enterprise capabilities through an AWS Marketplace listing, integrations such as AWS PrivateLink for security and privacy needs, and closer compatibility with customers’ existing AWS environments. Modal also plans to present its AI container technology at AWS re:Invent in December 2024.

OpenArt, a platform for AI image generation and editing, built complex image generation pipelines using proprietary ComfyUI workflows without compromising on customizability. However, they found that traditional model API providers were too inflexible for their needs. This led them to explore alternative solutions, including Modal, which provided a lightweight, programmatic solution that allowed them to deploy in minutes rather than hours. By leveraging Modal's autoscaling capabilities, OpenArt was able to scale their platform without overloading their tech stack, and can now power 100+ workflows on their platform.

The ComfyUI diffusion model platform has a rich custom node ecosystem, allowing developers to expand its capabilities through various nodes for image processing, animation, and usability. To help new users get started, the top 5 most used custom node packs across thousands of deployed workflows have been identified. These include the WAS Node Suite with hundreds of nodes, ComfyUI Impact Pack focused on image enhancement, ComfyUI IPAdapter Plus for style transfer, ComfyUI Essentials containing quality-of-life improvement nodes, and KJNodes for ComfyUI providing simple image transformation nodes. Each pack's description and use case have been provided along with sample code to run interactive ComfyUI sessions using a modal app or JSON workflow. The importance of ComfyUI Manager for discovering and installing custom nodes is also highlighted, as well as recent improvements in the ecosystem such as the Comfy Registry and serverless compute platforms like Modal.

Modal announced several platform updates, including static IP proxies for securely accessing private databases and resources, a native Slack integration for real-time app alerts, and a live dashboard for monitoring workspace resource usage. New GPU fallback options can expand function scheduling capacity, while client enhancements include shell access to running containers by ID, parameterized-function memory snapshots, sandbox output controls and IP allowlists, and ASGI lifespan protocol support. The company also shared its founder’s views on the evolving AI market and GPU demand, published examples for running and fine-tuning models such as Stable Diffusion 3.5, FLUX, Mochi, and Moshi, and added guidance for long training jobs and parallel hyperparameter sweeps. Modal further noted plans to attend AWS re:Invent and launch its YouTube channel, featuring a video on GPU-based hyperparameter sweep parallelization.

Modal has acquired Tidbyt, a New York-based maker of programmable LED smart-home displays, primarily to bring aboard its co-founders Rohan Singh and Mats Linander, whose backgrounds include container infrastructure work at Spotify and experience building large-scale systems. Tidbyt has shipped 100,000 consumer devices and developed expertise in hardware operations, customer acquisition, and business management, which Modal expects will complement its cloud infrastructure platform for data, AI, and machine-learning teams. Existing Tidbyt devices and services will continue operating, although no new devices will be shipped. Modal, whose customers include Suno and Substack, provides compute infrastructure for workloads such as GPU inference, LLM fine-tuning, and large batch processing.

Stable Diffusion 3.5 and Flux are two top text-to-image models, offering various output styles and features. Stable Diffusion 3.5 supports a wide range of output styles and has fast inference with its Large Turbo variant, making it suitable for both commercial and personal projects. Flux offers four model variants tailored for different users, including an open-source option, and is optimized for local and personal use. Both models have varying parameter sizes and GPU requirements, with Stable Diffusion 3.5 requiring a more powerful GPU for its larger variants.