September 2023 Summaries

There's a new open source LLM, Mistral 7B, which surpasses other models on benchmarks and has strong code-generation capabilities. It was released with an Apache 2.0 license and includes both a chat-tuned instruct variant and a base variant. The LLM's sliding 4k-token context window is an interesting new approach to attention during inference. Additionally, Baseten has introduced model observability features, including error codes, wake from sleep functionality, and a replica chart on the model metrics page. Furthermore, Baseten will host a series of in-person events exploring open source AI, starting with an event in New York and San Francisco.

The NVIDIA A10 and A100 GPUs are two popular choices for model inference tasks, including large language models like Llama 2 and Stable Diffusion. The A10 is a cost-effective choice capable of running many recent models, while the A100 is an inference powerhouse for large models, with higher performance in FP16 Tensor Core calculations. However, the A100 is also much more expensive to use, with a price per minute of $0.10240 compared to the A10's $0.02012. To balance latency and cost, users can consider using multiple GPUs in a single instance, such as combining 2-8 A10s or 1-8 A100s, which can also help run larger models like Llama 2-chat 13B. Ultimately, the choice between the A10 and A100 depends on the user's needs and budget, with the A10 offering a cost-effective alternative for many workloads.