First Impressions with LLaVA-1.5

Significant advancements in multi-modal language models have been made in 2023, with notable releases such as OpenAI's GPT-4(V)ision and Google's Bard. LLaVA-1.5, an open-source model, has emerged as a strong contender with its ability to handle text and image inputs, excelling in tasks like image description and visual question answering. Unlike GPT-4(V)ision, LLaVA-1.5 can be trained on a single 8-A100 GPU, making it more accessible. The model has demonstrated proficiency in zero-shot object detection and understanding unusual image contexts but has faced challenges with Optical Character Recognition (OCR), where it struggled with clear digital text and serial numbers. Despite its shortcomings, LLaVA-1.5's open-source nature and versatility highlight the rapid innovation in the field of multi-modal models, as researchers continue to explore the integration of text and image inputs for enhanced language models.