A 0.6B model at #2 on MTEB(Law)

Hanno-Labs reports that its 0.6-billion-parameter dinghy-law embedding model reached second place on the MTEB(Law) benchmark with a mean score of about 65.9, trailing only a much larger 7–8B specialist model and exceeding several larger open retrievers. Developed under limited time and compute using LoRA fine-tuning on a single GPU, the approach centered on measuring every change against a fixed baseline, selecting hard negatives through gradient alignment rather than embedding proximity, and manually reviewing training and evaluation examples to identify domain-coverage gaps. The authors found that nearest-neighbor negatives and synthetic perturbations could create false negatives, while hub filtering and gradient-selected, in-domain confusions improved results. Broader legal training data improved statute retrieval after shingle-based filtering was used to reduce evaluation leakage. To address catastrophic forgetting, the team applied WiSE-FT weight interpolation between the base and fine-tuned adapters, then averaged variants trained with and without mined negatives before merging them, while cautioning that hard negatives must be mined separately for each base-model size.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.