The Math Behind Inference for Llama3.1 405B

Yangqing Jia, CEO of Lepton AI, offers an insightful analysis of the economics behind AI inference APIs, particularly in the context of recent API offerings for Llama3.1 405B, emphasizing the often-overlooked role of both input and output tokens in pricing models. His analysis reveals that a Llama 405B model can achieve an output throughput of approximately 300 tokens per second with a concurrency of 10, generating potential revenue of about $798.34 per day based on Lepton's pricing of $2.8 per million tokens. Despite daily hardware costs of around $670.08 using AWS 8xH100 GPUs, Jia suggests profitability is achievable but with narrow margins and various influencing factors such as traffic variability, pricing models, and hardware costs. He highlights the importance of efficient operations, considering techniques like speculative decoding and prompt caching, and suggests that alternative GPU models might impact economic outcomes. Jia's analysis underscores the delicate balance companies must maintain between costs and optimizations to achieve profitability in the competitive AI API market.