April 2024 Summaries

In this article, three innovative benchmarks from CVPR 2024 are highlighted to showcase their potential impact on computer vision research. These benchmarks include ImageNet-D for testing the robustness of image classifiers against real-world perturbations, Polaris for assessing the ability of vision-language models to follow natural language instructions in interactive environments, and VBench for evaluating text-to-video generation models across multiple dimensions. Each benchmark presents unique challenges and opportunities for researchers, pushing the field towards more robust models. The article also discusses the design process, evaluation metrics, and potential impact of these benchmarks on future research directions.

The US Air Force Test Pilot School and DARPA have successfully installed AI agents in the X-62A VISTA aircraft as part of the Air Combat Evolution (ACE) program, demonstrating that AI can be used safely in aerospace applications. Microsoft has launched Phi-3 SLMs, which are smaller language models that compete with or outperform GPT 3.5 across all benchmarks. Microsoft Research project VASA-1 generates realistic talking faces in real-time based on audio input. According to Gartner's survey of over 500 business leaders, only 24% are currently using or piloting generative AI tools, with the majority still exploring or evaluating the technology. Microsoft has also released COCONut, a modernized large-scale segmentation dataset that improves upon COCO in terms of annotation quality and consistency across segmentation tasks.

Monocular depth estimation (MDE) is a critical task in computer vision, enabling depth prediction from a single image, which is essential for applications such as autonomous driving and robotics. This process is challenging due to the inherent ambiguity in projecting 3D scenes onto 2D images, requiring the consideration of cues like object size and perspective. The article explores the use of Hugging Face and FiftyOne for running and evaluating MDE models using the SUN-RGBD dataset. It highlights the use of transformer-based models like DPT and diffusion-based models such as Marigold, emphasizing the importance of visualizing depth maps beyond relying solely on evaluation metrics like RMSE, PSNR, and SSIM. The challenges of MDE, including data quality and the limitations of metrics in assessing model performance, are discussed, underscoring the necessity of a qualitative assessment of model predictions.

The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) is a significant platform for introducing new datasets and benchmarks in computer vision research. These datasets and benchmarks play a crucial role in advancing deep learning techniques by providing diverse, large-scale data and standardized tasks or challenges to evaluate model performance. Datasets are collections of data samples like images, videos, or annotations used to train and evaluate deep learning models. They provide the raw data that models learn from during the training process. Datasets often include labeled or annotated data to provide ground truth information for supervised learning tasks. Examples of famous datasets introduced at previous CVPRs include ImageNet (2009), Cityscapes Dataset (2016), Kinetics Dataset (2017), and nuScenes (2020). Benchmarks, on the other hand, are standardized tasks or challenges used to evaluate and compare the performance of different models or algorithms. They typically consist of a dataset, a well-defined evaluation metric, and a leaderboard ranking the performance of different models. Benchmarks allow researchers to gauge performance against state-of-the-art approaches and track progress in a specific task or domain. Examples include the 2009 Pedestrian Detection Benchmark and 2012’s KITTI Vision Benchmark Suite. The importance of datasets and benchmarks in deep learning cannot be overstated, as they serve as the foundation for training models and objectively assessing their performance. New datasets are essential to exploring novel tasks, accommodating emerging model architectures, and capturing diverse real-world scenarios. As models achieve higher – and eventually human-level – performance on established benchmarks, researchers create new, more challenging ones to push the boundaries of what's possible. At CVPR 2024, several new datasets and benchmarks have been introduced, each with the potential to advance progress in computer vision and deep learning. Three notable datasets are Panda-70M, 360+x, and TSP6K. These datasets cover a spectrum of computer vision tasks, including multimodal, multi-view, room layout, 3D/4D/6D, robotic perception datasets, and more. Panda-70M is a large-scale video captioning dataset that enables more effective pretraining of video-language models, driving progress on video understanding tasks. It overcomes the limitations of ASR annotations and provides a solution to manual annotations' cost and scalability issues. The 360+x dataset is a panoptic multimodal scene understanding dataset that covers multiple viewpoints (panoramic, third-person, egocentric) with multiple modalities (video, audio, location, text). It captures real-world complexity and provides a more realistic challenge for research. TSP6K is a dataset specifically designed for traffic scene parsing and understanding, focusing on semantic and instance segmentation in urban traffic scenes. These datasets address limitations in existing resources, provide rich and diverse data, and establish new performance benchmarks. They are poised to drive research forward and contribute to real-world applications like content creation, smart environments, and intelligent traffic management.

Detecting small objects in computer vision is challenging due to their diminutive size, limited information, and the inadequacy of standard datasets and models that focus on larger objects. Slicing Aided Hyper Inference (SAHI) is proposed as a solution, which involves dividing images into overlapping slices, applying object detection models to each slice, and merging the results for comprehensive detection across the entire image. This method, which is model-agnostic, leverages existing state-of-the-art models and enhances small object detection without requiring training on larger images. The technique increases the number of forward passes, leading to more predictions and improved recall and F1-score, especially for small objects, as evidenced by its application on the VisDrone dataset using YOLOv8. Despite the trade-off of increased computation, SAHI's ability to improve detection accuracy makes it a valuable tool in scenarios with many small objects, allowing for potential correction of missing ground truth labels through human-in-the-loop workflows.

The April '24 AI, Machine Learning, and Data Science Meetup recently took place, featuring presentations on resource-efficient text-to-image generative models, graph data patterns with knowledge graphs, optimizing training data, and exploring multimodal models. Maitreya Patel discussed ECLIPSE, a novel contrastive learning method for efficient T2I generation, while Andreas Kollegger presented GraphRAG, a model that uses Knowledge Graphs in RAG context. Mark Cox-Smith talked about optimizing training data with Voxel51 and V7 Darwin integration, and Harpreet Sahota demonstrated the use of fiftyone to evaluate LlaVa models on the TextQA dataset. Upcoming Meetups will take place in various locations worldwide, including Athens, Austin, Bangalore, Boston, Chicago, London, New York, Peninsula, San Francisco, Seattle, Silicon Valley, and Toronto.

The robotics industry is rapidly growing, with a global market value expected to reach $42.82 billion by 2024 and $65.59 billion by 2028. Computer vision plays a crucial role in various applications within the industry, including object detection and recognition, environmental monitoring and surveillance, visual inspection and quality control, and autonomous navigation. Key challenges faced by the robotics industry include safety standards and regulations, human-robot interaction, cybersecurity risks, cost, complexity, energy efficiency, and ethical and societal implications. Despite these challenges, the industry continues to thrive, driven by innovation in material science, new sensors, compute resources, AI, and computer vision.

The Conference on Computer Vision and Pattern Recognition (CVPR) is a premier annual event for presenting and discussing research in computer vision and pattern recognition. It brings together top academics and industry researchers to share cutting-edge work on object recognition, image segmentation, 3D reconstruction, and deep learning. In recent years, the conference has also focused on the intersection of computer vision and natural language processing. Five papers from CVPR 2024 that are set to redefine this intersection include: 1. "Describing Differences in Image Sets with Natural Language" by Lisa Dunlap et al., which presents VisDiff, a tool for efficiently describing differences between image sets using natural language. 2. "A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models" by Julio Silva-Rodríguez et al., which introduces CLAP, an efficient method for adapting large vision-language models to new tasks using only a few labelled samples. 3. "Let’s Think Outside the Box: Exploring Leap-of-Thought in Large Language Models with Creative Humor Generation" by Shanshan Zhong et al., which proposes the Creative Leap-of-Thought (CLoT) paradigm to enhance LLMs' ability to generate creative and humorous responses. 4. "Alpha-CLIP: A CLIP Model Focusing on Wherever You Want" by Zeyi Sun et al., which presents Alpha-CLIP, an advanced version of the CLIP model that improves its visual recognition ability while providing better control over the focus on image contents. 5. "mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration" by Qinghao Ye et al., which introduces mPLUG-Owl2, a modularized MLLM model that showcases modality collaboration in pure-text and multimodal scenarios. These papers demonstrate the ongoing commitment of the deep learning community to open science and innovation at the intersection of computer vision and natural language processing.

This week's FiftyOne tips and tricks blog covers various topics such as filtering images by type of class, computing mistakes in classification labels, grouping samples into a video, integrating FiftyOne with CVAT and Label Studio, and sorting detection patches by confidence. The community members share their questions and the experts provide detailed solutions to each query. These tips are helpful for users who want to improve their computer vision models using the open-source toolset, FiftyOne.

Stability AI has introduced Stable Audio 2.0, a new AI model that generates high-quality audio tracks from natural language prompts. McMaster and Stanford University researchers have created SyntheMol, a generative AI model that rapidly designs small-molecule antibiotics to fight antibiotic-resistant bacteria. Recent advancements in transformer-based models are improving time series forecasting capabilities across various industries. Gartner's recent survey found that only 24% of business leaders are currently using or piloting generative AI tools, with the majority still exploring or evaluating the technology.

Clustering, a fundamental unsupervised machine learning technique, is explored in the context of image data using tools like FiftyOne, Scikit-learn, and feature embeddings, such as those from the CLIP model. Unlike classification, clustering does not rely on predefined labels but rather uncovers inherent structures within data by grouping similar objects based on selected features. The article explains various clustering algorithms—centroid-based, density-based, and hierarchical—and emphasizes the importance of feature selection and dimensionality reduction techniques like UMAP to enhance clustering effectiveness. Using the FiftyOne platform, the blog demonstrates practical clustering applications, including visualizing clusters and labeling them with GPT-4V, highlighting the impact of different algorithms and hyperparameters. The discussion encourages experimenting with various models and parameters to optimally leverage clustering insights for better model training and data understanding, while also suggesting further exploration into different embedding models and clustering techniques for improved outcomes.

This week's tips and tricks blog covers various aspects of using FiftyOne, an open source machine learning toolset for computer vision models. Topics include integrating with popular annotation tools like CVAT, Label Studio, Labelbox, and V7; executing bash commands to save image files in a plugin; returning JSON when working with the AnnotationResults class; adding detector models to FiftyOne's inference functionality; colorizing classes in point cloud datasets; and more.

Voxel51 emphasizes the importance of good security as a high-empathy activity. ISO 27001 is an international gold standard for information security management systems, which helps protect data. The company considers this certification crucial to ensure data protection and build a secure organization and platform. For their community, ISO 27001 certification signifies enhanced data security, reduced risk of data breaches, and compliance with relevant regulations. Voxel51 encourages users to reach out for any questions or concerns regarding working securely with their data.