March 2024 Summaries

This week's FiftyOne tips and tricks blog covers various questions and answers from Slack, GitHub, Stack Overflow, and Reddit. Topics include performing semantic video search using the Semantic Video Search plugin, working with grouped datasets to represent multiview scenes, ensuring uniqueness of sample IDs within a dataset, specifying color schemes for dataset visualization, and configuring persistent dataset options. The blog also highlights FiftyOne's open-source community where everyone is welcome to ask questions and provide answers.

Ubisoft introduces "neo NPCs" like Bloom, designed to have meaningful conversations with players using generative AI, making game worlds more interactive and engaging. Apple is expected to address its approach to generative AI at the Worldwide Developers Conference (WWDC), exploring potential content partnerships and collaborations with leading AI entities. LLaVA-NeXT, an improved version of the LLaVA model, boasts enhanced reasoning, optical character recognition (OCR), and world knowledge capabilities. The latest advancements in training multimodal models are discussed by Chip Huyen, highlighting the importance of working with images and text to generate content and a more comprehensive understanding. Kate Park shares insights into the practical application and impact of data engines from her experience within the industry, emphasizing the need for diverse expertise in data labeling for tasks requiring deep domain knowledge.

RIOS Intelligent Machines, an end-to-end automation company, has successfully implemented FiftyOne Teams to efficiently organize and analyze its vast dataset of over 20TB. The platform seamlessly integrates with RIOS's existing ML experiment tracker, cloud data lake, and annotation tools, enabling a range of automated ML workflows. By automating data engineering tasks, the company has been able to eliminate unnecessary manual work and focus on improving its AI-powered robotic solutions for customers across various industries.

Voxel51 has released FiftyOne 0.23.7 and FiftyOne Teams 1.5.8, offering numerous enhancements and fixes for computer vision workflows. These updates include new features such as an Albumentations integration for data augmentation, added Places2 dataset to the zoo, a zero-shot image classification tutorial, and support for Python 3.11 and 3.12. Additionally, FiftyOne Teams extends FiftyOne with team collaboration capabilities on centralized data storage. The company encourages users to get involved in their open source community through various channels such as the FiftyOne Community Slack, Meetups, GitHub contributions, and industry spotlight blogs.

Deploying detection models is challenging due to various factors such as inference speed, camera lenses, and model selection. One critical issue is determining the confidence threshold for bounding boxes. The Optimal Confidence Plugin helps users avoid this problem by finding the best confidence threshold using F1 score instead of mAP (Mean Average Precision). This plugin considers true positives, false positives, and false negatives to determine the optimal threshold that maximizes model effectiveness during deployment while minimizing false positives. The Optimal Confidence Threshold plugin can be easily installed and used with FiftyOne datasets, improving workflows and enhancing model performance in real-world applications.

This week's FiftyOne tips and tricks blog covers various questions and answers from Slack, GitHub, Stack Overflow, and Reddit. Topics include loading saved views using Python SDK, troubleshooting MongoDB installation issues, computing area of instances in COCO format, exporting COCO predictions, and filtering samples based on a given field. The blog also provides an overview of FiftyOne, an open-source machine learning toolset that helps data science teams improve computer vision models by curating high-quality datasets, evaluating models, finding mistakes, visualizing embeddings, and getting to production faster.

The March '24 AI, Machine Learning and Data Science Meetup featured presentations on Visual Anagrams, Omnidirectional Computer Vision, Illuminating the Underground World with Multimodal Algorithms, and The Role of AI in Fixing Hiring. Andrew Owens discussed generating multi-view optical illusions using diffusion models, while Ciarán Eising covered omnidirectional cameras and their applications. Adonaí Vera explored multimodal algorithms for underground inspections, and Saurav Pandit examined the role of AI in mitigating hiring biases. The next Meetup will take place on April 18th with talks on video generation using space-time diffusion models and resource-efficient text-to-image generative models.

The Business Intelligence Group has named Voxel51's open source computer vision platform, FiftyOne, as a winner in its 2024 Artificial Intelligence Excellence Awards. This award recognizes companies leveraging AI technology to solve real-world problems. Launched in late 2020, the FiftyOne project has experienced significant growth and adoption across various industries. Its unique characteristics include an enthusiastic open source community, built-in flexibility and extensibility, and a plugin framework that allows users to extend and customize functionality.

NVIDIA GTC 2024 announced a new integration that enables autonomous vehicle (AV) developers to create, curate, and visualize robust synthetic training data using FiftyOne and NVIDIA Omniverse. The integration simplifies data generation and analysis, supports multi-sensor visualization, and converts data to common computer vision formats. It also allows for unstructured data exploration. This collaboration accelerates the path to autonomy in AV development by providing seamless data visualization and tailoring datasets for training and testing needs.

The article explores the integration of FiftyOne, an open-source toolkit for managing unstructured visual data, with MongoDB Atlas, focusing on the use of vector search to enhance data-centric workflows. FiftyOne provides an intuitive interface for working with diverse datasets, leveraging the non-relational capabilities of MongoDB to process images, videos, and more. The core functionality revolves around creating DatasetViews through flexible operations such as filtering and sorting, utilizing MongoDB aggregation pipelines for scalable data management. Vector search, facilitated by machine learning-generated embeddings, enables efficient indexing and querying of unstructured data, enhancing applications like image similarity searches and retrieval-augmented generation pipelines. The integration with MongoDB Atlas allows users to seamlessly combine traditional and vector search queries, optimizing efficiency and reducing the need for complex data processing tasks. The article highlights the ease of setting up vector search with features like the compute_similarity() method and the ability to perform semantic searches using natural language, thereby offering a powerful solution for visual data exploration and analysis.

This week's FiftyOne tips and tricks blog covers various questions and answers from Slack, GitHub, Stack Overflow, and Reddit. Topics include running inference only on unprocessed samples, deleting selected annotations, running multiple annotation jobs in CVAT, importing FiftyOne without binding to a port, and filtering and matching object classes. The blog also provides information about the open-source machine learning toolset, FiftyOne, which helps data science teams improve computer vision models by curating high-quality datasets, evaluating models, finding mistakes, visualizing embeddings, and getting to production faster.

The text discusses the use of unsupervised clustering in computer vision datasets, specifically through the FiftyOne Plugin called Interactive Clustering. It explains how clustering can be used to group similar samples together based on certain features or characteristics, and how it can help pre-label datasets, understand data landscape and density, and support baseline assumptions. The text also provides installation instructions for the plugin and demonstrates its use with examples from the KITTI dataset and MNIST dataset. It emphasizes the importance of being familiar with one's data when using clustering algorithms and suggests that users should always double-check results after automated labeling flows.

The year 2024 is expected to be a significant one for multimodal machine learning, with advancements in real-time text-to-image models and open-world vocabulary models. Contrastive language image pretraining (CLIP) has been at the heart of many of these advances since its introduction by OpenAI in 2021. CLIP aligns a vision encoder and a text encoder, enabling the model to understand both visual and natural language inputs. While OpenAI's CLIP model is well-known, there are other important data-centric advances in contrastive language-image pretraining that have improved upon its performance. These include ALIGN, K-LITE, OpenCLIP, MetaCLIP, and DFN. Each of these advances has contributed to the development of more effective multimodal machine learning models, with potential applications ranging from image classification to data filtering networks.

The integration between Hugging Face Transformers and the open-source FiftyOne library significantly enhances computer vision workflows by enabling the seamless application of transformer models to visual datasets. Transformers, initially designed for language modeling, have become pivotal in computer vision tasks such as image classification, semantic segmentation, and zero-shot inference, with the Vision Transformer (ViT) setting new performance standards. This collaboration allows users to apply transformer models directly to entire datasets or filtered subsets without custom coding, supporting tasks like image classification, object detection, and video inference. FiftyOne's capabilities for data curation and visualization, coupled with Hugging Face's diverse transformer models, provide a comprehensive and efficient framework for embedding computation, dimensionality reduction, and semantic similarity search. The integration reduces boilerplate code, aids in model comparison, and facilitates a deeper understanding of both data and models, highlighting the increasing impact of transformers in computer vision and multimodal machine learning.

Voxel51 has released FiftyOne 0.23.6 and FiftyOne Teams 1.5.7, offering numerous enhancements and fixes for computer vision workflows. These updates include new features such as tutorials on dimensionality reduction and data augmentation, formal Open CLIP integration, support for open-world object detection with YOLO World, and more. Additionally, the FiftyOne community offers various ways to get involved, including a Slack channel, Meetups, and contributions via GitHub.

This week's FiftyOne tips and tricks blog covers various questions and answers from Slack, GitHub, Stack Overflow, and Reddit. Topics include extracting SimilarityIndex, applying prediction labels to video samples, modifying sample fields in batches, rendering alpha matts, clipping video samples, and more. The community is open to all for asking and answering questions related to FiftyOne, an open-source machine learning toolset that helps data science teams improve computer vision models' performance by curating high-quality datasets, evaluating models, finding mistakes, visualizing embeddings, and getting to production faster.

The text discusses the importance of high-quality datasets in AI, as poor samples can negatively impact model performance. It introduces FiftyOne Plugins, a tool that helps identify and remove outliers from datasets. Outlier detection is demonstrated using embeddings and sklearn, with examples including finding classification and detection mistakes, removing duplicates, addressing image quality issues, and visualizing embeddings. The text also highlights the usefulness of the Outlier Detection Plugin in discovering unique samples that can be used for data curation decisions.

The security industry, valued at USD 143.07 billion in 2022, is projected to grow at a CAGR of 8.3%, reaching approximately USD 317.57 billion by 2032. Computer vision and AI have been introduced to bolster security systems with great success. Key applications include intruder detection, disaster search and rescue, security checkpoint screening, facial recognition, safety equipment recognition, among others. Companies at the forefront of computer vision in safety & security include Everon, Forsight, Vivint, Smiths Detection, Flyability, Protex AI, Canopy, Secury360, and MIDL Technology.

Data augmentation is a key technique in enhancing model performance by enlarging and diversifying training datasets, especially in fields like computer vision, where it helps mitigate overfitting and class imbalance. However, improper application can degrade model performance and lead to misinterpretations, as seen in examples involving wildlife conservation and medical imaging. The text emphasizes the importance of testing and understanding transformations before incorporating them into training pipelines, advocating for tools like FiftyOne and Albumentations to visualize and control augmentations. The article warns against treating data augmentation as a black-box process and highlights the significance of carefully curating transformations to maintain data quality and avoid undesired outcomes.

This week's FiftyOne tips and tricks blog covers various topics such as exporting and importing datasets, visualizing embeddings in FiftyOne, searching through videos inside the FiftyOne App, viewing images with multiple channels, and segmenting point clouds. The community members share their questions and receive helpful answers from other members or directly from the FiftyOne team. These tips provide valuable insights for users to effectively utilize the open-source machine learning toolset, enabling them to improve the performance of their computer vision models by curating high-quality datasets, evaluating models, finding mistakes, visualizing embeddings, and getting to production faster.