SlideOn & mSlideOn: Sparsified Late-Interaction and Dense Retrievers

LightOn introduces SlideOn and mSlideOn, open sparse retrieval models with 174 million and 339 million parameters that convert DenseOn and mLateOn representations into sparse vectors suitable for efficient inverted-index search. The models use sparse autoencoder pretraining to learn latent dictionaries, followed by supervised retrieval fine-tuning with contrastive learning, distillation, load balancing, and Matryoshka-style training that preserves performance when query and document vectors are pruned. On English BEIR-15 retrieval, SlideOn scores 56.68, exceeding its DenseOn base model and ranking strongly among sparse models below 3 billion parameters while offering adjustable quality, latency, and storage tradeoffs. mSlideOn performs particularly well on multilingual, long-document, and code retrieval, improving over mDenseOn by 4.76 points on full MIRACL multilingual evaluation and about 10.19 points on MLDR long-document retrieval, while reaching 74.78 on MTEB Code. The authors find that dense-model initialization works best for English retrieval, whereas late-interaction initialization with load balancing better supports multilingual and long-context tasks, and they release the models, autoencoder checkpoints, training data, and code under a permissive license.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.