September 2023 Summaries

This week's FiftyOne tips and tricks blog focuses on creating 3D detections in LIDAR or point cloud datasets. The article introduces the open source machine learning toolset, FiftyOne, which helps data science teams improve their computer vision models by curating high-quality datasets, evaluating models, finding mistakes, visualizing embeddings, and getting to production faster. 3D detection is a crucial aspect of computer vision in applications such as autonomous driving, robotics, and augmented reality. In FiftyOne, detections are defined with labels and bounding boxes for either 2D or 3D spaces. The article demonstrates how to create basic 3D bounding boxes, modify their location and dimensions, rotate them, and use 3D polylines. Additionally, it explains the importance of orthographic projections in creating thumbnails for LIDAR samples and improving dataset curation.

Accurate 3D bounding box annotation is crucial for building effective computer vision systems that utilize LiDAR and point cloud data, which are increasingly important in fields like autonomous driving, robotics, and spatial AI. This comprehensive guide delves into the creation and manipulation of 3D object detections using FiftyOne's open-source software, highlighting the key aspects of defining object location, rotation, and dimensions within a three-dimensional coordinate system. The tutorial explores various techniques, including constructing point cloud bounding boxes, handling irregularly shaped objects, and utilizing orthographic projections for better visualization and dataset curation. It emphasizes the importance of mastering these techniques to enhance dataset validation, model debugging, and annotation quality, ultimately allowing users to build scalable and accurate 3D lidar annotation pipelines for real-world applications.

This week's FiftyOne Plugin focuses on Zero-Shot Prediction, a process that generates initial predictions for computer vision tasks without explicitly being given any example predictions. The plugin unifies the interface across five tasks and allows users to specify label classes and models. It includes operators for zero-shot classification, object detection, instance segmentation, and semantic segmentation. Users can also choose to run inference on a subset of their data or delegate execution to be completed at a later time. The plugin is compatible with popular computer vision models like CLIP and SAM, as well as others that may be added by users.

The IEEE/CVF International Conference on Computer Vision (ICCV) is set to take place in Paris, featuring over 1000 papers discussing various aspects of computer vision. This article presents ten notable papers from the conference, including DEVA for open-world video segmentation, DWPose for whole-body pose estimation, EfficientViT and FastViT for efficient vision transformers, LightGlue for faster local feature matching, ProPainter for improved video inpainting, Segment Anything for prompted and unprompted segmentation tasks, Text2Room for extracting textured 3D meshes from 2D images, Text2Video-Zero for zero-shot text-to-video generation, and ViperGPT for visual reasoning via Python execution.

This tutorial explores the use of FiftyOne and Ultralytics YOLOv5 for training drone data, a task that is becoming increasingly important as drones revolutionize industries from agriculture to surveillance. The guide covers key steps such as curating high-quality datasets, evaluating models, visualizing embeddings, and getting to production faster. It also highlights the importance of considering factors like angle, height, and lens type when collecting data for drone use cases. By leveraging these powerful tools, users can navigate the complexities of object detection training and make informed decisions in the dynamic intersection of drone technology and artificial intelligence.

This week's FiftyOne tips and tricks blog delves into the use of polylines in computer vision tasks. Polylines are a sequence of connected line segments used to represent and approximate the shape of an object or region within an image, commonly employed in tasks such as image annotation, object tracking, and shape analysis. The blog covers creating classic polylines, polygons with polylines, cuboids, and rotated bounding boxes using FiftyOne's implementation of polylines. These diverse labeling techniques enable computer vision professionals to more effectively and precisely address a wide array of real-world problems, facilitating better representation and understanding of complex visual data.

Polyline annotation is a crucial technique in computer vision that offers more precise object representation than traditional axis-aligned bounding boxes, enhancing the accuracy and flexibility of image annotation workflows. By utilizing FiftyOne, an open-source tool, users can implement polylines to track object contours, define regions of interest, and build sophisticated annotation workflows, including polygons, rotated bounding boxes, and 3D cuboids. These methods are particularly beneficial for applications in autonomous driving, object tracking, OCR, robotics, and spatial reasoning systems, where precise annotation directly influences model performance. Polylines, cuboids, and rotated bounding boxes are indispensable tools for accurately capturing the spatial dimensions and orientations of objects, thereby improving object detection and tracking across various domains.

In week five of "Ten Weeks of Plugins", a series dedicated to building FiftyOne plugins, we discuss Optical Character Recognition (OCR) and Keyword Search. The PyTesseract OCR plugin leverages the Tesseract OCR engine to perform optical character recognition on samples in a dataset, while the Keyword Search plugin allows users to search within labels generated by the first plugin. These two plugins combined enable searching through documents like pages of old books, handwritten notes or resumes based on their textual content.

The nuScenes dataset is a comprehensive resource for autonomous driving computer vision research. It consists of multi-modal sensor data, including high-resolution camera images, LiDAR point clouds, and RADAR sweeps. With meticulous labeling of objects such as vehicles, pedestrians, and cyclists in each frame, nuScenes enables the development of highly accurate computer vision models for autonomous vehicles. The dataset provides a broad spectrum of real-world driving scenarios, ensuring that AI systems can adapt to the unpredictability of the open road. By integrating nuScenes into the FiftyOne computer vision toolset, researchers and engineers can efficiently explore and analyze this valuable resource, pushing the boundaries of autonomous vehicle technology.

FiftyOne Teams 1.4 has been released, introducing significant upgrades to its representational power and extensibility. The new features include Dataset Versioning for tracking data lineage, Delegated Operations for scheduling customizable tasks from within the App, and native Ultralytics integration for easy training of YOLOv8 models on FiftyOne datasets. These enhancements aim to make FiftyOne Teams a more flexible and efficient tool for organizing, visualizing, and iterating on data in AI workflows.

Researchers from Friedrich-Alexander-Universität Erlangen-Nürnberg in Germany have developed a new dataset, CaFFe, to improve the mapping of glacial boundaries known as calving fronts. Calving is an important natural phenomenon that contributes significantly to sea level rise and climate change modeling. The dataset consists of 681 SAR images of marine-terminating glaciers in the Arctic, Greenland, and Antarctic ice sheets taken from multiple satellites over two decades. By providing long-term calving front information using SAR imagery from multiple satellites, CaFFe introduces new challenges such as different penetration depths or sensitivity to surface changes; different signal-to-noise ratios; and different geometries, topographic effects, shading, and overlay effects.

Human pose estimation is a crucial component of computer vision, enabling machines to understand spatial relationships and dynamics from visual data by detecting anatomical features within images and videos. This technique leverages advanced deep learning models, such as CNNs and transformer-based architectures, to address challenges like occlusion and motion blur. Applications of pose estimation span robotics, sports analytics, healthcare, and augmented reality, making it a key area of AI research. The tutorial on pose estimation with FiftyOne and CVAT guides users through creating and annotating pose skeletons using keypoint-based annotations, facilitating efficient dataset preparation and annotation workflows. FiftyOne's integration with CVAT allows for streamlined skeleton annotation, making the process accessible to both novices and experienced practitioners, thereby enhancing computer vision and machine learning projects.

The September 14th Computer Vision Meetup featured presentations on ARMBench, an object-centric benchmark dataset for robotic manipulation; Visualizing Defects in Amazon’s ARMBench Using Embeddings and OpenAI’s CLIP; From Model to the Edge, Putting Your Model into Production; and Optimizing Distributed Fine-Tuning Workloads for Stable Diffusion with the Intel Extension for PyTorch on AWS. The event also included Q&A sessions and discussions about various aspects of computer vision technology. Upcoming Meetups will cover topics such as Automatic Prompt Optimization, Glacier Monitoring with Computer Vision Models, and Building Natural Language Applications with txtai. The Computer Vision Meetup network is sponsored by Voxel51, the company behind the open source FiftyOne computer vision toolset.

This week's FiftyOne tips and tricks blog focuses on creating a skeleton dataset using keypoints. The blog introduces FiftyOne, an open source machine learning toolset that helps data science teams improve computer vision models by curating high-quality datasets, evaluating models, finding mistakes, visualizing embeddings, and getting to production faster. Pose skeletons are vital for understanding human or animal motion in images or videos, facilitating identification of precise body position and movement annotation. In FiftyOne, pose skeletons are stored with the Keypoints class. The blog demonstrates how to create a view or dataset of images, tag them, prepare the dataset to expect keypoint skeletons, and annotate the skeleton using FiftyOne's CVAT integration. Finally, it explains how to load the annotations back into the dataset for further analysis.

This week's FiftyOne Plugin focuses on Image Deduplication, a task that removes exact and approximate duplicate images from a dataset. Duplicate data can lead to longer training times, higher costs, and lower performance in machine learning models. The plugin allows users to find both exact and approximate duplicate images, visualize these groups of duplicates, and delete all duplicates or keep a representative from each set of duplicates without writing any code. It includes eight operators for deduplication workflows and supports native integrations with Pinecone, Qdrant, Milvus, and LanceDB for large datasets.

The FACET (FAirness in Computer Vision EvaluaTion) benchmark dataset has been released by a team at Meta to study and evaluate the fairness of computer vision models. It is designed to be comprehensive, diverse, and representative of various demographics, genders, and skin tones. The dataset includes 31,702 images with 49,551 unique person detections, spanning protected attributes, unprotected attributes, lighting conditions, and more. FACET aims to ensure that models do not learn biases by providing evaluation metrics for detection and classification predictions based on concepts and attributes. This dataset can be used to assess model disparity and make necessary improvements in building equitable computer vision models.

This week's blog delves into dynamic grouped datasets in FiftyOne, an open source machine learning toolset that helps data science teams improve computer vision models by curating high-quality datasets, evaluating models, finding mistakes, visualizing embeddings, and getting to production faster. Dynamic grouping is a feature in FiftyOne that allows users to group samples in their dataset by a particular field or expression. The blog provides examples of how to use dynamic groups for organizing video frames and working with dynamically grouped views. It also discusses the benefits of using dynamic groups, such as bringing clarity to data relationships and speeding up workflows.

The AI, Machine Learning, and Data Science Meetup held on September 7, 2023 featured presentations on Neural Residual Radiance Fields for Streamably Free-Viewpoint Videos, EgoSchema: A Dataset for Truly Long-Form Video Understanding, and Monitoring Large Language Models (LLMs) in Production. The event also included Q&A sessions with the speakers. In lieu of swag, attendees had the opportunity to vote on a charity to receive a $200 donation, which was awarded to Coalition for Rainforest Nations. Upcoming Meetups will take place on September 14 and beyond, featuring various topics related to AI, machine learning, and data science. The Computer Vision Meetup network is sponsored by Voxel51, the company behind the open source FiftyOne computer vision toolset.

In this monthly blog series, we celebrate recent milestones and happenings within the FiftyOne community. A highlight from the open source community is Aidence, a company that uses FiftyOne to improve healthcare diagnostics and treatments by providing practical and intuitive AI solutions for physicians. Community members are encouraged to share their success stories using FiftyOne and claim rewards. In August, there were point releases for both open-source and Teams versions of FiftyOne. Three new computer vision model integrations were added: Segment Anything, DINOv2, and PyTorch Hub Integration. The FiftyOne GitHub repository has over 4,100 stars, 72 contributors, and is used by 379 organizations. Last month, the community welcomed almost 150 new members to the Slack channel. Upcoming events include Computer Vision Meetups and workshops. New resources such as blogs, videos, tutorials, and cheat sheets are available for users to improve their skills with FiftyOne.

In week three of "Ten Weeks of Plugins", a YouTube Player Panel Plugin was built. This plugin is a joint Python/JavaScript plugin with a single operator, open_youtube_player_panel, which opens a custom JavaScript component named YouTubePlayerPanel. The user can choose to play a video from a preset list or enter a URL. The plugin showcases the basic elements required to build a panel in FiftyOne using JavaScript and React Material UI components.

This week's FiftyOne tips and tricks blog delves into Grouped Datasets. Grouped datasets are collections of multiple slices of samples, possibly of different modalities (image, video, or point cloud), organized into groups. They can be used to represent multiview scenes where data for multiple perspectives of the same scene can be stored, visualized, and queried in ways that respect the relationships between the slices of data. To create a grouped dataset, first import FiftyOne and define your groups. Then prepare the data by creating a dictionary of filepaths and their corresponding groups. Afterwards, add the samples to the dataset using the `add_samples` function. Finally, visualize the dataset with the `launch_app` function. Working with grouped datasets is similar to working with non-grouped datasets. You can access basic information about your dataset and change the active slice. To get the entire group for a sample, use the group ID and pull like normal. Iterating through your grouped dataset can be done in two ways: iterating through your active slice or iterating through each group using the `iter_groups` function. Creating views in your grouped dataset is also possible, allowing you to sort, slice, and search through your dataset. You can filter based on class, exclude individual groups from a view, create views of joined group slices, and perform aggregations. The possibilities are endless with FiftyOne's powerful grouped datasets feature.

Grouped datasets in FiftyOne's open-source platform offer a robust method for organizing and analyzing related samples across various modalities or perspectives, such as multi-camera images, synchronized sensor streams, and multimodal machine learning workflows. These datasets maintain relationships between samples while utilizing FiftyOne's dataset view and querying system to its full potential. The article provides a comprehensive guide on creating and managing grouped datasets, demonstrating how to define group slices, iterate through grouped samples, and build expressive dataset views. It covers advanced querying patterns, filtering, and aggregations that are particularly beneficial for large-scale computer vision workflows. By leveraging FiftyOne's features, users can efficiently manage and analyze complex multiview datasets, facilitating a scalable approach to computer vision and multimodal machine learning pipelines. The walkthrough concludes by encouraging users to apply these techniques to their computer vision projects and outlines future tutorials for exploring dynamic grouped datasets and advanced customization techniques.