UltraBERT: One Encoder for Classification and Retrieval

UltraBERT is an Apache 2.0–licensed family of English encoder models from DMIS Lab, including 150M- and 380M-parameter backbones with native 32K-token context windows and a 150M embedding variant. Based on extensive training experiments, the authors report that encoder downstream performance saturates after roughly 1–2 trillion tokens despite continuing loss reductions, and that low masking ratios favor classification while high ratios favor retrieval. UltraBERT addresses this trade-off by training a shared encoder with separate language-model heads for 5% and 50% masking, applying distillation only to the low-masking branch, which the authors say enables strong performance in both task types. Reported results place the models first on 9 of 10 evaluated benchmarks at their respective sizes, with the 380M model reaching 91.0 on GLUE and 47.7 on BEIR while retaining 99% passkey accuracy through 32K tokens. The accompanying UltraBERT-Embed-150M model, trained through contrastive learning without larger-embedder distillation, reportedly achieves a 67.24 score on MTEB English v2 and is intended for dense retrieval of text, code, long documents, and reasoning-oriented queries.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.