Encoding the World's Medical Knowledge into 970K

The article introduces the BiomedBERT Hash series of models, which are compact AI models designed for medical applications on devices with limited computational power. These models, including the biomedbert-hash-nano with only 970K parameters, leverage a modified embeddings layer to encode medical knowledge efficiently, showing performance competitive with much larger models. The article discusses the training and evaluation of various models, such as cross-encoder and ColBERT models, highlighting their effectiveness in tasks like semantic search and text classification using datasets from PubMed. Distillation and fine-tuning techniques are employed to optimize these models, resulting in high performance despite their small size. NeuML, the creator of these models, also offers AI consulting services and is working on hosting solutions for txtai applications.