September 2024 Summaries

Baseten has partnered with Google Cloud to provide high-performance AI infrastructure through the Google Cloud Marketplace, making it easier for organizations to build and scale AI applications. This collaboration enables seamless integration of Baseten's platform within existing cloud environments, offering secure data management and efficient deployment of AI models. The partnership addresses the growing demand for scalable, reliable AI infrastructure that works smoothly with current cloud solutions.

Compound AI systems integrate multiple AI models and processing steps to form a cohesive workflow capable of handling complex tasks. These multi-step processes can introduce high latency and performance bottlenecks in production applications. Using MongoDB Atlas and Baseten’s Chains framework for compound AI, developers can build high-performance compound AI systems like RAG that can scale to handle massive production traffic without introducing bottlenecks. By combining MongoDB Atlas Vector Store for data retrieval and Baseten for model inference, developers can create scalable, secure, performant compound AI applications.

NVIDIA has announced support for function calling and structured output for LLMs deployed with its TensorRT-LLM Engine Builder, adding model server level support for two key features. Function calling allows users to pass a set of defined tools to an LLM as part of the request body, while structured output enforces an output schema defined as part of the LLM input. These features are built into NVIDIA's customized version of Triton inference server and use logit biasing to ensure valid tokens are generated during LLM inference. The implementation has minimal latency impact after the first call with a given schema is completed, allowing for efficient use of these new features.

A new feature has been introduced in TensorRT-LLM Engine Builder to generate structured output during LLM inference. This includes JSON mode, where model output matches a given JSON schema, and function calling, where the LLM selects from provided tools to accomplish a task. Both functionalities have no marginal impact on tokens per second and are available for all LLMs deployed using the Engine Builder. The new features aim to address challenges in integrating LLMs with structured data, enabling developers to call LLMs with guaranteed output structure while adding negligible latency.