October 2024 Summaries

Cerebrium now supports ASGI (Asynchronous Server Gateway Interface) applications, which enables efficient handling of multiple concurrent connections in Python web applications. This support allows users to build and deploy real-time streaming applications, complex ML pipelines, and intuitive dashboards while maintaining cost efficiency and performance. Users can also batch process requests for better cost management. Deploying an ASGI FastAPI application on the Cerebrium platform is quick and easy, with a single command.

Cerebrium now offers ASGI (Asynchronous Server Gateway Interface) support, addressing common MLOps challenges related to model deployment, scalability, and real-time processing. ASGI enables efficient handling of multiple concurrent connections, allowing users to build real-time streaming applications, complex ML pipelines, and intuitive dashboards with improved performance and cost efficiency. Applications benefit from ultra-low latency communication within the Cerebrium cluster, facilitating instant updates and efficient resource management. The platform simplifies deploying ASGI FastAPI applications with examples and detailed configuration guides, while offering support and credits to help ML startups manage costs. Users can explore further resources and join the Cerebrium community for ongoing deployment support and updates.

Cerebrium, in collaboration with Rime, has developed an alternative solution to OpenAI's Realtime API that combines large language models (LLMs) and voice capabilities. The partnership leverages the Pipecat framework from Daily to deliver an efficient and customizable solution for enterprises and developers. This approach offers enhanced performance, cost savings, and flexibility compared to OpenAI's offering. By using OpenAI's GPT-4 as the LLM and Rime for Text-to-Speech model, users can achieve similar latency with a 3x cost saving.

OpenAI's recent launch of the Realtime API has generated significant interest by merging large language models (LLMs) with voice capabilities, yet Cerebrium and Rime propose an advanced alternative that surpasses OpenAI's offering in performance, cost, and flexibility. The collaboration employs the Pipecat framework from Daily to integrate LLM and voice solutions, allowing for seamless, real-time voice processing with AI functionalities. The platform's setup facilitates efficient transcription and turn detection, accommodating responsive AI assistants that can switch between services to optimize performance and cost. The solution offers a robust and customizable platform for enterprises, showcasing superior speed and affordability over OpenAI's API, and also promises further customization and scalability opportunities.

In this tutorial, Michael Louis from Cerebrium benchmarked vLLM, SGLang, and TensorRT for Llama 3.1 API on a single H100 GPU. The goal was to compare Time To First Token (TTFT) and throughput across various batch sizes. Results showed that vLLM had the lowest TTFT of 123ms, while SGLang achieved the highest throughput of 460 tokens per second on a batch size of 64. The choice of framework depends on user constraints and preferences for either low-latency or high-throughput applications.

Cerebrium conducted a benchmark test on the LLaMA 3.1 70B FP8 model across three popular frameworks—vLLM, TensorRT, and SGLang—focusing on Time To First Token (TTFT) and throughput using a single H100 GPU. vLLM emerged as the best for low-latency applications with a TTFT of 123ms due to its "token stream" approach, which reduces latency by overlapping computation and communication. TensorRT, leveraging mixed-precision and quantization techniques, offers efficient inference for large language models, while SGLang, with its dynamic workload distribution and GPU optimization, excelled in throughput, achieving 460 tokens per second at a batch size of 64. The benchmark highlights that the choice of framework depends on specific needs, whether prioritizing latency or throughput, and suggests configurations such as multi-GPU setups for optimal performance.