De-mystifying Multimodal Learning: The Hidden Inefficiency in Vision Language Modelling

In the transition from text-only models to Vision Language Models (VLMs), the concept of Visual Tokens (VT) emerges as a crucial factor influencing performance and feasibility. The text explores the mathematical and operational complexities of calculating VTs across various state-of-the-art strategies, such as Qwen's dynamic merging, LLaVA's Any-Resolution grids, and Gemma3's Pan&Scan approach. These methods address the inefficiencies of fixed-resolution models like LLaVA-1.5 by adapting to native image resolutions or employing dynamic grid splitting, albeit with varying computational costs and token efficiencies. The study highlights the importance of understanding VT calculations to optimize VLM deployment, emphasizing that mastering this aspect is essential for efficiently leveraging multimodal systems.