Latency, the Ultimate Gen AI Constraint

Latency is a significant constraint for AI assistant tools, particularly when using third-party large language models (LLMs) for autocomplete suggestions, as this can result in delays that affect usability, as evidenced by Sourcegraph Cody's experience. The latency issues arise from the complex process of collecting context, making model inferences, and merging results, which is exacerbated when relying on third-party APIs due to additional network delays and potential rate limiting. To address latency without altering model characteristics, developers have implemented solutions such as smart model compilation, optimized model architectures, model parallelism, and smart batching, which help reduce inference time by leveraging GPU resources more efficiently. These innovations allow for faster processing of larger models, enhancing the quality of AI-generated suggestions while accommodating a high number of concurrent users. The challenge is further compounded by the rise of open-source models, which tempt companies to build in-house solutions, often underestimating the necessary infrastructure and facing performance limitations.