August 2024 Summaries

A1111 is an open-source web UI for Stable Diffusion, created in August 2022, which offers a beginner-friendly interface with built-in common workflows such as text-to-image and image-to-image. However, its reliability has been criticized and it lacks cutting-edge features like Flux. On the other hand, ComfyUI, launched in January 2023, is a more powerful and flexible version of A1111, allowing users to define workflows down to the node level in a flowchart-like UI and offering more flexibility and customizability. With its strong momentum and pace of development, ComfyUI is likely more future-proof, but also has a steeper learning curve. Both tools can be used simultaneously for different purposes, with A1111 suitable for rapid prototyping and ComfyUI for serving complex workflows in production. Running both on Modal's serverless infrastructure allows users to avoid installation headaches and only pay for GPU time used.

The fine-tuning of large language models (LLMs) is a computationally expensive process, but new techniques such as LoRA and QLoRA have made it more efficient by reducing the number of parameters to update. LoRA, or Low-Rank Adaptation, involves freezing pre-trained weights and training smaller "adapter" matrices that represent the update to the base model, which requires significantly less VRAM than full fine-tuning. In contrast, QLoRA, or Quantized LoRA, further reduces memory usage by quantizing the low-rank matrices, achieving a 4x reduction in memory usage compared to standard LoRA. While both techniques can lead to a loss of knowledge, QLoRA's quantization may actually reduce overfitting. When it comes to choosing between LoRA and QLoRA, the decision depends on available hardware resources, with LoRA being recommended for models that fit within 16GB VRAM, while QLoRA is suitable for smaller devices or those with limited space.

At Modal, the goal is to make running code in the cloud as intuitive and easy as running code locally. To achieve this, they've developed a system that spins up cloud-based containers quickly, ideally under 1 second. The key to achieving this involves understanding containers, shortening image pulls by reducing image bloat and avoiding unnecessary overhead like Docker, caching frequently accessed files locally, and implementing content-addressed caching. By focusing on what's essential for running an application and leveraging these techniques, Modal has significantly reduced the time it takes to start containers, making it easier for developers to deploy and scale their applications in the cloud.

At Modal, a new interactive code playground has been introduced to make running code in the cloud super fast and simple. The playground allows anyone to write and run code on Modal directly from their web browser without installing it. It includes interactive tutorials for new users and is designed to be a convenient place for experienced users to explore new features and experiment with code. The playground uses Modal Sandboxes, which provide an isolated environment to safely execute arbitrary user code. When a user hits the "Run" button, the frontend sends an HTTP request to the web server, which authenticates the request and executes the program in the sandbox using the user's ephemeral token credentials. The standard output and standard error logs are streamed back to the server. The playground is designed to help users explore and experiment with Modal, but it has limitations, such as not being able to access all of Modal's features without installing the Python package and setting up Modal locally.

The cost of an NVIDIA H100 GPU can range from around $25,000 per GPU for a direct purchase to over $400,000 for a full system. However, alternatives like GPU-on-demand platforms offer flexible access to high-performance GPUs without significant upfront investment, with prices varying between $2.99 and $9.984 per hour depending on the platform. The total cost of using H100 GPUs also depends on factors such as cold start time, model loading time, inference speed, and input/output operations, which can vary significantly depending on the platform used. These serverless GPU platforms provide a more accessible alternative to direct purchase, allowing users to leverage the power of H100s without the hefty upfront investment, making them an attractive option for organizations looking to build AI applications with ease.

When OpenAI open-sourced Whisper, a great speech-to-text model was provided but it lacked some key features such as speaker diarization and word-level timestamps. To address these gaps, various Whisper variants were developed, including WhisperX, which adds automatic speaker recognition and speed, making it ideal for multi-speaker transcriptions; Whisper JAX, which offers extreme speed on TPU v4 hardware; Whisper.cpp, a lightweight C++ implementation that allows edge device usage; Distil-Whisper, a smaller and faster version of Whisper; and Whisper Streaming, a real-time transcription model. Ultimately, the best choice depends on specific needs such as accuracy, speaker identification, scalability, or offline processing, with WhisperX being recommended for its balance of ease-of-use and performance.

Modal is reducing prices for CPUs and selected high-performance GPUs by 15–30% in response to falling GPU market prices driven by increased supply and newer hardware, aiming to maintain competitive usage-based pricing for AI developers. It is also revising CPU and memory billing so each running container is charged according to the greater of its reserved or actual resource use, while establishing a default minimum reservation of 0.125 CPU cores, a change that may raise costs for low-utilization long-running CPU jobs. The company recommends using concurrent inputs to improve CPU utilization in affected workloads. Modal’s Team plan will increase to $250 per month, adding unlimited seats and $100 in monthly credits alongside recently introduced platform capabilities such as region selection, custom domains, WebSockets, and cloud bucket mounts. The new pricing applies immediately to new users and many Starter users, while Starter users facing higher CPU bills and Team or Enterprise customers will transition in September unless they request early enrollment.

Richard Sutton’s “bitter lesson” is presented as supporting not only larger-model learning but also inference-time search, which can improve smaller models by generating and evaluating many candidate outputs. In experiments using LLaMA 3.1 8B on the HumanEval Python programming benchmark, researchers used test suites as objective evaluators and found that increasing generations from one to 100 raised pass@k performance from 66.4% to 90.5%, roughly matching GPT-4o’s reported 90.2% pass@1 score, while 1,000 generations reached 95.1%. The work, run with Modal serverless GPUs, vLLM batching and caching, and isolated code-execution sandboxes for testing, reportedly cost under $50 and demonstrated predictable scaling across several orders of magnitude. The results support the broader claim that, in domains with fast and reliable outcome evaluation such as coding and formal mathematics, search can trade additional compute for reduced model-memory requirements and enable smaller open models to approach or exceed larger systems, although applying this method to subjective or open-ended language tasks remains difficult.