February 2023 Summaries

Truss is an open-source model packaging format designed to improve the developer experience for serving ML models by enabling live reload, which allows for iterative development without a long upload-build-deploy loop. The feature requires simplicity of use, persistence, identical behavior to published production models, self-sufficiency, no increase in model build time, and recovery from mistakes. Truss achieves this through a proxy server called the Control Proxy, which is decoupled from the inference server and uses a check-and-set mechanism to guard against race conditions. The feature also supports live reload on local Docker and remotely deployed models on Baseten, with patch computation and application being crucial components of its operation.