Why Whisper cuts off Indic transcripts after six seconds

Adalat AI faced an issue with its Whisper transcription system, which truncated Indic language transcripts after six seconds due to a tokenization limitation. This problem arose because Whisper, a transformer model, is capped at generating 448 tokens, with half reserved for previous text prompts, leading to a 224-token limit for new text. Indic languages, unlike English, are tokenized inefficiently, with each character often taking up three tokens due to a lack of byte-level merges. This inefficiency caused transcripts to end abruptly, marked by a replacement character, when the cap was reached mid-character. Adalat AI addressed this by shortening Voice Activity Detection (VAD) windows and modifying their CTranslate2 fork to allow up to 448 new tokens, improving transcript accuracy. However, the root issue lies in the tokenizer's bias towards English due to its training data, underscoring the need for more diverse language corpora and potentially script-aware tokenizers to improve fairness in machine learning models.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.