December 2023 Summaries

Voxel51 celebrated its fifth anniversary in 2023, marking a significant milestone for the company that focuses on open-source computer vision tools. The year was marked by several key developments in the field of computer vision, including GPT-4V, which enables users to instruct GPT-4 to analyze image inputs; SAM, a promptable segmentation system with zero-shot generalization capabilities; and DINOv2, a self-supervised system that can learn from any collection of images. Other notable developments included YOLO-NAS, an object detection model that outperforms competitors in terms of accuracy-speed performance; SDXL and SDXL Turbo, which enable near real-time generation of high-quality images; LoRA, a technique for parameter-efficient fine-tuning; DALL-E 3, an image generation model integrated with ChatGPT; Runway Gen-2, a multimodal AI system that can generate novel videos from text, images or video clips; Pika Labs, a startup developing an AI-powered platform for video editing and generation; and Emu Video and Emu Edit, which streamline the processes of training T2V models and precisely editing images via text prompts.

In 2023, computer vision made significant progress across various modalities. Notable developments include YOLO-NAS for object detection, the Segment Anything Model (SAM) for segmentation, DINOv2 for self-supervised learning, Gaussian Splatting as an alternative to NeRFs, and advancements in text-to-image models like Midjourney and Stable Diffusion. Additionally, LoRA facilitated efficient fine-tuning of diffusion models, while the Ego-Exo4D dataset emerged as a foundation for video perception research. Furthermore, T2V models made strides towards high-quality video generation from text prompts, and multimodal LLMs like GPT-4 Vision and LLaVA combined language understanding with visual capabilities. Finally, LLM-aided visual reasoning enabled the integration of general reasoning abilities with expert vision models for tasks such as visual question answering.

Voxel51 and V7 have partnered to integrate their data-centric AI tools, empowering customers to build high-quality dataset management pipelines. The integration between Voxel51's FiftyOne and V7 Darwin allows users to curate datasets, perform annotation review and QA, evaluate model performance, and transfer data seamlessly between the two platforms. This partnership aims to improve AI products by ensuring top-tier quality in dataset management and annotation processes.

LLaVA (Large Language and Vision Assistant) is an open-source project developed by researchers at the University of Wisconsin, Microsoft Research, and Columbia University. It aims to create a novel end-to-end trained large multimodal model that can compete against even the giants of models such as GPT-4. The LLaVA team created 150k image-instruction pairs using images from the COCO Train2017 dataset and leveraged GPT-4 to form conversations about the image in a cheap and efficient manner. They used the widely popular CLIP VIT-L/14 visual encoder model and Vicuna, an LLM based on Llama 2, for their model training. The results show that LLaVA was able to capture overall an 85% relative score compared to GPT-4. The dataset has been updated to include more datasets to train on other than COCO, bringing in over 665K conversations now.

The thirty-seventh Conference on Neural Information Processing Systems (NeurIPS) took place in New Orleans, LA from December 10th to 16th, 2023. With over 3,584 accepted papers and numerous tutorials and workshops, the conference covered a wide range of topics in machine learning. Among these were advances in multimodal machine learning, which include techniques that enable machines to process and understand information from multiple types of data simultaneously, such as text, images, and audio. The ten most notable papers on multimodal machine learning at NeurIPS 2023 are: 1. Chameleon: This paper presents a technique for extending pure language models to challenging multimodal tasks by leveraging the general reasoning capabilities of large language models in conjunction with tools for web search, mathematical analysis, and visual understanding. 2. Cheap and Quick: The authors introduce a new technique called Mixture-of-Modality Adaptation (MMA) for fine-tuning vision-language models more efficiently by training lightweight adapters instead of the entire model. 3. DataComp: This paper presents a competition and benchmark for evaluating novel filtering strategies for constructing multimodal datasets, as well as a state-of-the-art 1 billion sample multimodal dataset. 4. Holistic Evaluation of Text-to-Image Models (HEIM): The authors propose the first holistic evaluation benchmark for text-to-image models, which encapsulates 12 aspects of performance and incorporates both computational metrics and crowd-sourced human evaluations. 5. ImageReward: This paper presents a general-purpose reward model for human preferences on T2I generated images, as well as an approach for aligning T2I models with these preferences using Reinforcement Learning from Human Feedback (RLHF). 6. InstructBLIP: The authors apply instruction tuning to vision-language models and present a family of state-of-the-art zero-shot models, as well as fine-tuned models that achieve SOTA on specific tasks. 7. LAMM: This paper presents a dataset and benchmark for evaluating 2D and 3D visual reasoning in multimodal large language models. 8. MagicBrush: The authors present a high-quality hand-crafted image editing dataset, which includes both single-turn and multi-turn edits, and demonstrate its utility in improving performance on instruction-guided image editing tasks. 9. OBELICS: This paper presents the first web-scale dataset of natural multimodal documents with interleaved images and text, and demonstrates its utility in training a new 80 billion parameter model that is competitive with Flamingo. 10. Pick-a-Pic: The authors present a large-scale dataset of prompts, generated images, and human preferences for T2I models, which can be used to train models to better align with user preferences.

The thirty-seventh Conference on Neural Information Processing Systems (NeurIPS) will take place from December 10th-16th, 2023 in New Orleans. With 3,584 accepted papers, researchers worldwide will discuss various topics such as Bayesian optimization and adversarial attacks. An analysis of the abstracts reveals that "generative" (508), "transformer" (271), and "agent" (280) are popular terms, while "foundation" (111) is less common than "efficient" (963). The number of authors per paper has increased over the past decade, with an average of 4.98 in 2023. Titles have also grown longer, from an average of 7.39 words in 2017 to 8.72 in 2023.

The Dec 7, 2023 AI, Machine Learning and Data Science Meetup covered various topics such as building private search systems with Retrieval Augmentation using Haystack, the rise of intelligent data platforms, and scaling similarity search with USearch. During the event, attendees had the opportunity to vote for their favorite charity, The Coalition for Rainforest Nation (CfRN), which received a $200 donation on behalf of the computer vision community. The Meetup also provided information about upcoming events and ways to get involved in the Computer Vision Meetups network.

Voxel51 has released general availability updates for its open-source machine learning toolset, FiftyOneOne. The latest versions, FiftyOneOne 0.23 and FiftyOne Teams 1.5, bring enhancements and fixes to streamline computer vision workflows. Key features include new integrations with Redis and MongoDB for native text and image searches on datasets, as well as additional collaboration tools in FiftyOne Teams. The updates also improve the overall user experience and functionality of the platform.

In this article, the author shares their experience creating over 25 GitHub repositories in a year. They emphasize the importance of an effective README file as it serves as the first impression for potential users. The author provides tips and tricks on using GIFs, links, and tables to enhance the README's content and user experience. Additionally, they discuss tools like Gifox and XConvert for creating and compressing GIFs, as well as techniques for linking within a repository and formatting tables in Markdown. The author also shares how to automate table updates using Python scripts and GitHub Actions.

Stability AI has introduced Stable Video Diffusion, a new way to experience Generative Artificial Intelligence (GenAI). This tool allows users to animate images by passing them into the model. The plugin can be used with FiftyOne datasets through a new image-to-video plugin. To use it, install the plugin and set up a replicate account. Then, load a dataset in the app, select samples to convert into videos, and execute the "img2video" operator. Results will be saved in a new dataset called "image2video." The plugin is addictive and fun to experiment with different images.