Model2Vec: Distill a Small Fast Model from any Sentence Transformer

Model2Vec is an innovative technique designed to create a smaller, faster, and high-performing static model from any Sentence Transformer by leveraging methods like Principal Component Analysis (PCA) and Zipf weighting. This approach significantly reduces the dimensionality of token embeddings and optimizes their weighting, enabling it to deliver fast, hardware-efficient, and eco-friendly embeddings without the need for large datasets. Despite being uncontextualized, Model2Vec maintains strong performance across various tasks, often outperforming older models like GloVe and BPEmb and showing comparable results to models like MiniLM on specific tasks. Ideal for applications requiring rapid and lightweight embeddings, Model2Vec can be easily integrated into existing pipelines that support Sentence Transformers, offering both distillation and inference modes. Ablation studies underscore the importance of using Sentence Transformers, PCA, and Zipf weighting for achieving optimal performance, making Model2Vec a compelling choice for text classification, clustering, and other natural language processing tasks.