Tokenization is Killing our Multilingual LLM Dream

Omar Kamali discusses the significant challenges faced by low-resource languages in training large language models (LLMs), particularly focusing on the issue of tokenization. Kamali highlights how tokenization, the process of converting text into numerical data for LLMs, often fails to capture the intricacies of morphologically rich and low-resource languages, leading to poor performance. He notes that while creating language-specific tokenizers can be beneficial, it disrupts cross-lingual alignment and fails to encompass the full diversity of language use, especially with typos and variations. The article suggests that current tokenization models are inadequate, as they demand considerable computational resources without offering significant gains in understanding. Kamali proposes exploring continuous pre-tokenization layers as a potential solution, which could allow LLMs to process text as a continuous signal rather than discrete tokens, potentially improving the model's ability to handle multilingual inputs without sacrificing performance.