LLaVA: Advancing Vision-Language Models Through Visual Instruction Tuning

LLaVA (Large Language and Vision Assistant) is a pioneering effort to implement text-based instruction for visual-based models, combining large language models with visual processing capabilities. It uses pre-trained LLMs like Vicuna to process textual instructions and the visual encoder from pre-trained CLIP, a ViT model, to process image information. LLaVA is fine-tuned on multimodal instruction-following data generated using GPT-4 or ChatGPT, enabling it to perform tasks like summarizing visual content, extracting information from images, and answering questions about visual data. The evaluation results demonstrate the effectiveness of visual instruction tuning, as LLaVA's performance consistently outperforms two other visual-based models: BLIP-2 and OpenFlamingo.