January 2023 Summaries

The improved logs experience on Baseten provides better visibility into common issues such as out of memory (OOM) errors, enabling faster root cause diagnosis during model deployment and invocation. This is achieved by separating build logs from deployment and prediction logs, matching the two-step process of deploying a model on Baseten. The platform also offers improved resources management and instance sizing, with guides available to help users set the right resources for their models and avoid unnecessary overhead. Additionally, Truss 0.2.0 has simplified its developer experience, removing friction from model packaging, while a new version-first model management UI enables better model version management, allowing users to deploy multiple models, update models based on new data or to combat model drift, and review version statuses at a glance.

Scaling your ML model horizontally can help handle high traffic, but it's not just about adding more replicas and relying on autoscaling to manage the load. There are key considerations to keep in mind, such as handling variable demand, managing infrastructure costs, and ensuring even utilization of resources. By understanding these limitations and using a combination of techniques like response caching and model optimization, you can optimize your ML model's performance and reduce waste.

When deploying a packaged machine learning model to a cloud service like AWS, choosing the right instance size is crucial to handle your model while minimizing compute cost. This decision involves two key factors: CPU or GPU and memory size. Models can be served on either a CPU or a GPU, with GPUs being more powerful but also more expensive. If your model can run on a GPU and invocation speed matters, select an instance with an attached GPU. Otherwise, stick with less expensive CPU instances. The second decision is selecting the appropriate memory size, which should be based on the size of your model weights files and other necessary files to have loaded into memory. By considering these two factors, you can select the most suitable instance type for your model deployment while minimizing costs.