July 2024 Summaries

Baseten is a platform that helps engineering and machine learning teams deliver their best work by providing delightful products for production inference in AI-native startups and enterprises. The company has a strong market opportunity, with AI inference being a massive and rapidly growing field. Baseten's customers are early adopters serving early adopters, but as the market grows and matures, the company will benefit from mass adoption. The platform is open-source and works with high-quality models for audio transcription, chat, image generation, and more. Baseten exists to solve hard problems for smart teams, enabling them to deliver their best work and redefine what generative AI is capable of. The company has real customers, revenue, and an unbelievable market opportunity, and it's entering a critical phase of growth where every person who joins now can have a massive impact on the direction and magnitude of its growth. Baseten offers competitive salaries, stock grants, great benefits, and access to the latest GPU hardware, making it a great place for people to grow their skills and advance in their careers. The company also prioritizes collaboration, camaraderie, and investing in employees, with epic offsites featuring show-stopping food, all-hands strategy meetings, and a highly competitive hackathon.

The new feature in Baseten allows users to deploy custom ComfyUI workflows as APIs, enabling easier integration and deployment of ComfyUI models. The `build_commands` feature enables the packaging of custom nodes and model checkpoints within a ComfyUI workflow, reducing cold start times for container builds. Users can now easily modify existing ComfyUI workflows to an API-compatible format using handlebars templates, and deploy them as production-grade API services on Baseten. This new feature aims to provide an excellent developer experience around ComfyUI deployments.

You can now serve thousands of fine-tuned LLMs from a single GPU using LoRA (Low-Rank Adaption) swapping with TensorRT-LLM on Baseten, which maintains low time to first token (TTFT) and high tokens per second (TPS). This allows for efficient inference and model management, making it feasible to serve multiple fine-tuned models from a single deployment. LoRA swapping is compatible with in-flight batching, does not affect latency significantly, and can handle thousands of active and cached LoRAs. The implementation involves caching LoRAs on system memory or GPU VRAM, with load times ranging from instant to 2 milliseconds depending on storage location. The method also allows for flexible inference by specifying which fine-tune to use through a three-part format that includes task_id, weights, and config parameters.

Baseten's asynchronous inference allows for smooth processing of long-running requests, spikes in traffic, and request prioritization, reducing timeouts and improving GPU utilization. This method adds requests to a queue based on model capacity and priority, ensuring that tasks don't overwhelm the model and allowing for more efficient use of resources. It provides visibility and control over requests, enabling developers to track status, cancel requests as needed, and access results through webhooks or cloud storage, making it a robust solution for handling long-running jobs and spikes in traffic.

Baseten Chains is a Python framework designed to build and orchestrate multi-component AI workflows with optimal scaling for each component. It enables developers to combine business logic with ML models, making model pipelines easy, performant, and robust. Chainlets are modular services linked together to form a full workflow, which can be composed using control flow and dependencies. Chains supports local testing and debug execution, hybrid testing modes, and remote deployment on Baseten, ensuring coherence across the overall application. The framework aims to address pain points from Truss, such as code organization, error detection, and tooling for local execution, while providing a holistic developer experience with modularized inference pipelines.