SraVaani: How Vision Helps Hearing

SraVaani is a multilingual Indian automatic speech recognition model from ARTPARK-IISc that supports 65 languages and dialects and is built using the Vaani corpus, which contains more than 31,000 hours of speech across 105 languages but relatively limited transcription coverage. Its central approach adds an audio-image alignment stage between self-supervised audio pretraining and supervised ASR fine-tuning, exploiting the corpus’s picture-prompt collection method to teach a FastConformer speech encoder to associate spoken descriptions with related image embeddings without requiring new transcripts. The model uses a 17-layer FastConformer encoder, frozen SigLIP2 vision representations during alignment, and a hybrid CTC-TDT decoder for final speech recognition training on 30,565 hours of labeled speech from 18 public datasets. On a held-out Vaani evaluation spanning 48 languages, the alignment method reduced word error rate from 28.09% to 27.40%, with reported improvements in 37 languages and particularly strong coverage for lower-resource tribal and dialect languages where comparison systems often produced no output. SraVaani-1.0 is released through Hugging Face in NeMo format, alongside training and evaluation resources for adaptation to other speech datasets.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.