September 2024 Summaries

Serverless GPUs are cloud computing services that allow users to run GPU-accelerated workloads on-demand, without managing underlying hardware or software. They offer a cost-effective way for developers to deploy and scale their AI models, video processing tasks, and other GPU-intensive applications. Several new serverless GPU providers have emerged, including Modal, RunPod, Baseten, and Replicate, each offering unique features and use cases such as model serving, fine-tuning, and training, as well as CI/CD pipelines. These platforms provide flexible deployment options, pre-trained models, and private endpoints for users to deploy and interact with their GPU-accelerated applications.

RabbitMQ is a traditional message-oriented middleware that excels in low-latency message queuing and routing, making it suitable for complex routing scenarios, traditional publish-subscribe messaging, and microservices communication. It offers flexible routing, multiple protocols, push model, and message acknowledgment features. In contrast, Apache Kafka is a distributed event streaming platform designed for high-throughput, fault-tolerant, and scalable messaging, ideal for high-throughput event streaming, log aggregation, stream processing, and event sourcing. Kafka excels in handling millions of messages per second, maintaining consistent low latency even at scale, and offering massive horizontal scalability, while RabbitMQ is optimized for lower throughputs and can be scaled horizontally but not to the same extent as Kafka. Both systems provide robust fault tolerance, high availability, message persistence, and durability features, with Kafka optimized for long-term storage and high-throughput scenarios. The choice between RabbitMQ and Kafka depends on specific use cases, with RabbitMQ suitable for traditional messaging patterns and low-latency requirements, and Kafka ideal for high-throughput event streaming and large-scale data processing needs.

Google Cloud offers two primary serverless computing options: Cloud Run and Cloud Run Functions. Cloud Run is a container-based, fully managed serverless platform that allows developers to deploy and scale containerized applications with minimal infrastructure management. It offers features such as flexible scaling, fine-grained billing, HTTP/2 and HTTPS support, custom domains, concurrency, GPU support, VPC connectivity, and event-driven capabilities. In contrast, Cloud Run Functions is a simplified function-as-a-service that focuses on single-purpose functions written in supported languages, primarily designed for event-driven architectures with automatic scaling, simplified development, and focus solely on writing function code without container management. The key differences between the two services include deployment models, abstraction levels, container management, use cases, and pricing models. Cloud Run is suitable for longer-running services, web applications, and complex architectures, while Cloud Run Functions is ideal for short-lived, event-based actions and simple API endpoints. By understanding these differences, developers can choose the right service for their project.

Google Cloud Run functions offer a flexible and cost-effective serverless solution for running your code in the cloud. The pricing model for both first generation and second generation functions is based on different components, including compute time, memory usage, invocations, and networking. First generation functions are priced per GHz-second and GB-seconds, while second generation functions are priced per vCPU-second and GB-seconds. Google Cloud offers a generous free tier for both generations of functions, with varying limits and requirements. Understanding the pricing components and how they are calculated is crucial to optimizing costs and choosing the right function type for your workload.

Serverless inference is a cloud computing model that allows for deploying and serving machine learning models without managing underlying infrastructure. It offers several advantages, particularly for expensive transformer-based models, including cost-efficiency, scalability, reduced operational overhead, and flexibility. Serverless inference eliminates idle GPU time costs, automatically scales to handle varying loads, and reduces the need for manual server management. While it may appear more expensive on a per-minute basis compared to traditional deployments, it can lead to significant cost savings for workloads with variable demand. Various cloud providers offer serverless capabilities, including Google Cloud Run Functions, which supports running GPUs, but is currently in preview. To optimize serverless inference deployments, it's essential to leverage GPU acceleration, minimize cold starts, optimize model loading and initialization, implement efficient batching, and follow best practices for each of these aspects.

AWS Lambda and Google Cloud Functions are two popular serverless execution environments for building and connecting cloud services. Both offer rapid development and scalable applications by allowing developers to write single-purpose functions triggered by events or HTTP requests. A comparison of these two services reveals key differences in features, performance, pricing, and integration capabilities. AWS Lambda supports a wider range of languages and offers native VPC integration, while Google Cloud Functions provides deeper integration with Google Cloud services. Pricing models differ, with AWS Lambda charging in 1ms increments after the first 100ms and Google Cloud Functions charging in 100ms increments. Both services offer robust event-driven architectures and cold start mitigation strategies, including provisioned concurrency and minimum instances. GPU support is available for both platforms, but with different limitations. VPC support is essential for security and access to private resources, with AWS Lambda offering native integration and Google Cloud Functions providing serverless VPC Access Connector feature. Ultimately, the choice between AWS Lambda and Google Cloud Functions depends on specific use cases and requirements.

Apache Airflow and Dagster are two popular data orchestration tools used to build and manage complex data pipelines. Airflow is a highly flexible, open-source workflow management system known for its ease of use and strong community support. It uses Python to define workflows as Directed Acyclic Graphs (DAGs), allowing users to schedule, monitor, and manage complex data pipelines. Airflow's key strengths include its massive ecosystem of plugins and integrations, cloud-native design, web-based UI, high flexibility, and customizability. Dagster, on the other hand, focuses heavily on data quality, testing, and analytics, with built-in data quality checks, a strong focus on testing and debugging workflows, and an asset-centric approach to data pipelines. Dagster is particularly well-suited for analytics-focused tasks, such as collecting data from APIs, processing and transforming data, visualizing results, and emphasizing metadata and data source information. When choosing between the two tools, consider Airflow if you need a highly flexible and customizable workflow management system or want to leverage its vast ecosystem of plugins and integrations, while considering Dagster if your primary focus is on data quality and testing throughout the pipeline. Both tools cater to different needs and preferences, making them powerful choices for data orchestration.

AI agents are generative tools that autonomously perform tasks given natural language prompting. Devin, a software engineering AI agent, is well-known for its sophisticated reasoning and UI, while open-source alternatives like OpenHands, SWE-agent, Aider, smol-developer, and Devika offer similar capabilities with varying degrees of research focus and complexity. These agents can be used to automate tasks such as code generation, troubleshooting, and issue resolution, and some have raised seed rounds to support their development and deployment.

Contextual AI, a company offering an end-to-end platform for building enterprise AI applications, faced challenges with their continuous integration (CI) process due to the need for flexible resources. They used tiny models as stand-ins during large-scale tests and had to manually run tests on in-house GPU nodes, which was inconvenient and time-consuming. After searching for alternative CI providers, they found Modal, a platform that offers flexible configurations of GPUs on-demand, allowing them to automate their test suite and maximize developer iteration speed while maintaining a high quality bar.

Llama 3.1 is Meta's latest family of large language models that are quickly becoming the standard in the open-source LLM space, available in three sizes and a fine-tuned Instruct version optimized for instructions and dialogue. Serving Llama 3.1 as an API requires significant compute, especially with the 405B version, but can be done on Modal's serverless compute platform using GPUs like A100s and H100s while only paying for what you use. The process involves creating a Modal account, cloning the examples repo, and adjusting GPU VRAM settings accordingly. Pricing is usage-based, with automatic spinning down and scaling in production. Llama 3.1 offers a generous community license, making it a great choice for fine-tuning and serving as a commercial product. With open-source serving framework vLLM and Modal's compute platform, building a Llama 3.1 API for production-grade LLM inference is easy at a cost-effective price point.

Jupyter notebooks have evolved into powerful platforms for complex data analysis and AI development, requiring significant computational power. Traditional approaches of having dedicated GPU workstations or clusters come with challenges such as high upfront costs, underutilization during off-peak times, difficulty in scaling resources up or down based on project needs, and maintenance headaches. However, a serverless platform like Modal's provides flexible on-demand GPU resources, eliminating these challenges by allowing users to spin up and down resources as needed. With the installation of the Modal Python package and authentication with the platform, users can launch Jupyter notebooks backed by Modal GPUs in seconds, providing a seamless and scalable solution for demanding computations.

Grado is an open-source Python library that enables developers to create interactive web interfaces for their machine learning (ML) models with minimal code. It simplifies the process of making ML models accessible and user-friendly, supporting various input and output types. Deploying Gradio in the cloud offers several benefits, including rapid prototyping, easy integration with popular ML frameworks, customizable UI components, and serverless architecture. Modal provides a platform that makes deploying Gradio interfaces in the cloud straightforward, allowing developers to fine-tune their ML models and store their weights on the platform. By combining Gradio and Modal, developers can create intuitive web interfaces for their ML models, deploy them quickly and efficiently, and collaborate with others more effectively.

Ollama is an open-source project that simplifies the process of running and managing large language models. It offers a powerful command-line interface, allowing users to easily integrate it into their workflows and run multiple models on the fly without requiring a daemon restart. Ollama provides access to a wide range of pre-configured models and can be integrated with serverless cloud computing platforms like Modal, which leverages GPU resources for improved performance. To use Ollama on Modal, users need to have an account at modal.com, install the Modal Python package, authenticate with the Modal CLI, and then run a specific command that deploys the Ollama service on Modal and runs an inference with their specified text. The code for Ollama is organized into a systemd service configuration file and a main application code that defines key components such as model and pull functions, creates a Modal image, and encapsulates the Ollama functionality in a class.

The XTTS (eXtended Text-to-Speech) model is a high-quality open-source text-to-speech system that offers multilingual speech synthesis capabilities. To run XTTS using Modal, a serverless cloud computing platform, users need to create an account at modal.com, install the Modal Python package, and authenticate their account. The script uses a single Python file to set up and run XTTS, importing necessary libraries and setting up the Modal app, defining the image that will be used to run the model, and implementing the XTTS class with methods for loading the model and speaking text. The script also defines an entrypoint function to run the XTTS model, taking a text input and saving the output as a WAV file. To use this script, users need to save it into a file, run it using Modal, and provide the text to be converted to speech.

ChatTTS is a high-quality open-source text-to-speech library that can be integrated into applications. To run ChatTTS using Modal, a serverless cloud computing platform, developers need to create an account at modal.com, install the Modal Python package, and authenticate their account. They then set up the environment by importing required libraries, configuring an image with necessary dependencies, creating a TTS class that handles text-to-speech conversion, and defining a local entrypoint to run the conversion. The script can be saved in a file named chattts_modal.py and run using Modal, generating an output.wav file with synthesized speech.

Meta's Llama3-405B is a large language model that represents a new frontier in open-source models, offering capabilities rivaling top closed-source AI models. However, its size and computational requirements make it daunting to run. To overcome this, the guide outlines optimizations such as 8-bit quantization, multi-GPU setup, and reduced VRAM footprint. The process involves creating an account at modal.com, installing the Modal Python package, authenticating the account, and using three separate files from the provided gist: downloading the model weights, setting up the vLLM server, and interacting with the model. The guide also provides options for customization when interacting with the model, such as adjusting generation parameters or providing a custom prompt.

AWS SAM is an open-source framework that allows developers to define their entire serverless stack, including Amazon API Gateway APIs, AWS Lambda functions, and Amazon DynamoDB tables, using a single YAML file. This approach offers several advantages such as infrastructure as code, local testing, simplified deployment, version control friendly, CI/CD integration, and more. To get started with SAM, developers need to install the AWS CLI and SAM CLI, initialize a new project, explore and modify the code, and deploy the function using the `sam deploy --guided` command. By following these steps, developers can easily deploy their Lambda functions using SAM and take advantage of its benefits for serverless development and deployment workflows.

We've been using this system ourselves and are enjoying: Moving millions of rows of data a day for less than $1 a day. Clean code built on open source frameworks with strong community support.

We've developed a serverless GPU cloud platform called Modal that offers fast and developer-friendly application development and deployment, particularly for machine learning prediction inference functions. Our platform uses a lazy loading approach to improve efficiency and responsiveness by initializing components as they are needed. This allows us to load the BERT model, which is typically around 512 MiB in size, in just 200 milliseconds from disk cache and about 300 milliseconds from the network. We've achieved this through various optimizations such as caching, using big hosts with sufficient bandwidth, optimizing FUSE settings like read-ahead and request sizes, managing congestion and background threads, and minimizing heap usage.

Batch processing and stream processing are two distinct approaches to data processing, differing in when the data is processed. Batch processing involves collecting data over time and processing it in large chunks at scheduled intervals, often ideal for handling large volumes of data where immediate results are not critical. In contrast, stream processing handles each data item in real-time as it arrives, suitable for scenarios requiring immediate data processing and analysis. The choice between batch and stream processing depends on the specific use case, with batch processing traditionally used for handling large volumes of data and stream processing used for real-time applications. Various technologies, including message brokers, streaming frameworks, and streaming databases, have emerged to support both approaches.

This guide provides a step-by-step process for creating an AWS API Gateway endpoint that uploads JPEG images to Amazon S3 using AWS Lambda and TypeScript. The solution involves setting up a new directory, initializing it with necessary dependencies, creating a Lambda function in TypeScript that checks for file content and filename parameters, verifies the image type, and uploads the file to S3, and then building and deploying a SAM template that includes an API Gateway endpoint that triggers the Lambda function. Once deployed, users can test their API using cURL by sending a POST request with the image file and filename parameter. The solution provides a scalable and efficient way to handle image uploads in serverless applications.

The guide discusses the challenges of fine-tuning Large Language Models (LLMs) due to GPU memory constraints, particularly VRAM bottlenecks. A general rule of thumb for full fine-tuning with 16-bit precision is 16GB of GPU memory per 1 billion parameters in the model. For a 7B parameter model, the estimated total VRAM requirements are approximately 70GB when using half-precision and 8-bit optimizers. Techniques like LoRA (Low-Rank Adaptation) and QLoRA (Quantized LoRA) significantly reduce VRAM requirements by up to 80% in some cases, making efficient fine-tuning possible for larger models. The guide provides a comparison table of VRAM requirements for different model sizes and fine-tuning techniques, highlighting the importance of considering VRAM constraints when training LLMs.

A rule of thumb for large language models is approximately 2GB of GPU memory per 1 billion parameters in the model, which can help estimate the required GPU memory. When loading a model in "half precision" (16-bit), this ratio increases to around 140GB for a 70B model, indicating that a single A100 80GB GPU may not be enough but two A100 GPUs could suffice. Quantization reduces the amount of GPU memory needed by reducing the precision of the model's weights, with common levels including 16-bit (half-precision), 8-bit, and 4-bit. The formula M = (P x (Q/8)) x 1.2 can be used to calculate the required GPU memory for a model with quantization, considering the number of parameters, bits used for loading the model, and an additional 20% overhead for tasks like key-value caching.