June 2024 Summaries

Runpod, in collaboration with Defined.ai, launched a pilot program to democratize AI development by providing startups access to enterprise-grade datasets and affordable compute resources, which are typically cost-prohibitive. This initiative allowed selected developers, like Theseus AI, to access specialized datasets, such as conversational speech data, to train state-of-the-art models. Specifically, Theseus AI aimed to improve its Automatic Speech Recognition (ASR) model for the financial sector, where transcription precision is crucial. Utilizing Runpod's high-performance cloud platform, Theseus AI successfully fine-tuned its Whisper model on Defined.ai's financial-specific audio data, dramatically reducing the Word Error Rate (WER) from 18% to 1.7% on validation datasets. This collaboration underscored the importance of sector-specific data in achieving advanced model performance and highlighted the potential of accessible, high-quality computational resources to level the playing field for AI startups, fostering innovation and bridging the data wealth gap.

Runpod's serverless offering now supports multiple GPUs, enhancing its capability to run large language models (LLMs) with ease. Users can assign two A100 or H100 GPUs or up to ten 24GB or 48GB GPUs to a worker, facilitating the execution of 70 billion parameter models at full precision or nearly any quantized model using the VLLM Quick Deploy template. Setting up involves creating a network volume to store models, reducing cold start times to approximately 600ms for models like Llama-3-70b. Serverless architecture, while requiring more initial setup, offers cost efficiency by billing only for active use and allowing dynamic scaling to handle concurrent requests, providing a smoother user experience compared to fixed pod setups.