January 2024 Summaries

Embeddings, high-dimensional numeric vectors representing input data features, are crucial in fields like computer vision and large language models, yet their complexity poses challenges for human interpretation. Dimensionality reduction techniques such as PCA, t-SNE, and UMAP address this by translating high-dimensional data into a more interpretable lower-dimensional space, each with unique strengths and weaknesses. PCA is efficient and straightforward but assumes linear relationships, while t-SNE excels at preserving local structure in nonlinear data but struggles with scalability. UMAP, which balances local and global structure, offers better scalability than t-SNE but depends on randomness and hyperparameters. These methods, illustrated using the CIFAR-10 dataset and models like ResNet-101 and CLIP, provide varied insights into data visualization. Additionally, custom methods, like Isomap and CompressionVAE, offer alternative approaches for dimensionality reduction, emphasizing the importance of choosing the right technique based on the specific data structure to gain meaningful insights.

The AI, Machine Learning and Data Science Meetup recently took place on January 25, 2024, featuring presentations on various topics such as the SANPO dataset, Setmlvis tool for object detection model comparison, segmentation capabilities of the Segment Anything Model (SAM) on fisheye images, and a lightning talk about Storia Lab, an AI-based image/video editor. The event also included Q&A sessions where attendees could engage with speakers to gain deeper insights into their work. Additionally, the Meetup announced its upcoming schedule for February 15, featuring talks on fair computer vision, food waste classification with AI, objects and image geo-localization from visual data, and next-generation video understanding. The event is sponsored by Voxel51, a company that develops open source tools like FiftyOne to improve the performance of computer vision models.

Voxel51 has released FiftyOneOne 0.23.3 and FiftyOne Teams 1.5.4, which bring numerous enhancements and fixes for computer vision workflows. The updates include native Hugging Face and SuperGradients integrations, a delete selected labels operator, a monocular depth estimation tutorial, API optimization, and optimized cloud media export. These releases contain over 30 new features and improvements.

Video understanding is an important yet complex area of study in computer vision due to its multiple modalities, time-series elements, open-ended questions, and answers. With the increasing amount of video data available, AI models that can parse through millions of videos to understand if it falls into a set search criteria are crucial for efficient data processing, training, or storage. Two main approaches to understanding videos are Vision Question and Answering (VQA) and Action Recognition. VQA LLMs have vast knowledge of human language and context in images but can only work on frames or single image-based prompts. On the other hand, action recognition models can take in video and understand the time component of data but lack depth in responses and are limited to a set number of classified actions. For general understanding of video datasets, VQA is currently considered more effective than action recognition models. However, the evolution of Video Understanding remains an unsolved problem with many potential approaches being explored.

The global sports market is growing rapidly, with a projected value of $512.14 billion in 2023. Computer vision and artificial intelligence (AI) are playing an increasingly important role in the industry, enabling real-time video analysis, fitness and health tracking, sports predictions, and improving the overall fan experience. Key challenges facing the industry include player health and performance, fan engagement in the digital age, and venue evolution. Some exciting applications of computer vision in sports include sports analytics and strategy, injury prevention and rehabilitation, AI referee assistance, and enhancing the fan experience through augmented reality (AR) and virtual reality (VR). Companies at the forefront of this technology include the United States Tennis Association (USTA), AiSport, Hawk-Eye Innovations, Sportlogiq, Ludimos, and Tonal.

The article explores the development of a semantic search engine specifically for emojis, addressing the limitations of existing emoji search tools that rely solely on exact text matches. The author outlines the challenges faced in creating an engine that integrates both textual and visual data, highlighting the unique dual nature of emojis as both text and images. By utilizing models like CLIP and generating high-resolution images, the search engine attempts to bridge the gap between these modalities for more effective retrieval. The process involves generating candidate emojis based on image similarity, reordering them through text-based similarity evaluations, and using a cross-encoder for refined rankings. The final result is an open-source emoji search engine available in UI and CLI versions, which, despite its imperfections, represents a step forward in multimodal search technology. The article emphasizes the broader implications of understanding the intersection of text and images, with emojis serving as a unique test case for multimodal models.