October 2023 Summaries

The text discusses various aspects of the open source computer vision toolkit, FiftyOne. It mentions upcoming workshops on FiftyOne Plugins and Getting Started with FiftyOne. Topics covered in these workshops include creating Python and JavaScript plugins, using annotation API to add or edit labels, working with video datasets, grouping images and videos together, auditing and versioning data, evaluating ground truth annotations, and support for point clouds and instance segmentation.

This week's FiftyOne tips and tricks blog covers various topics such as running the FiftyOne App inside a Sagemaker instance, leveraging GPUs to compute embeddings, avoiding duplicates in train and test datasets, converting image-sized masks to bounding box relative masks, exporting FiftyOne datasets except for images, and more. The blog also encourages readers to join the FiftyOne community on Slack, GitHub, Stack Overflow, and Reddit.

The text discusses the creation of a plugin-builder/manager plugin, which is a Python plugin with four operators that streamline plugin workflows. These operators allow users to manage existing plugins, install new ones, construct UI components for input, and generate scaffolding for new Python operators or plugins. The plugin aims to make plugin management and development easier by outsourcing boilerplate code so developers can focus on the core logic. Community contributions are encouraged to improve this plugin further.

This blog post discusses how to use the Segment Anything Model (SAM) for predictions and segmentations on Kaggle's Football Player Segmentation dataset. The post provides an overview of FiftyOne, a machine learning toolset that helps data science teams improve computer vision models by curating high-quality datasets, evaluating models, finding mistakes, visualizing embeddings, and getting to production faster. It also explains the purpose of the Football Player Segmentation dataset and how it can be used for player detection and segmentation in football matches. The post then delves into using SAM with FiftyOne, including installation requirements, loading datasets, adding embeddings, filtering data, working with SAM, and evaluating predictions.

The Voxel51 engineering team has released FiftyOneOne 0.22.2 and FiftyOne Teams 1.4.3, which bring numerous enhancements and fixes to streamline computer vision workflows. These updates include improvements in the FiftyOne App, Core, Plugins, and Zoo components, as well as general enhancements and bugfixes for FiftyOne Teams. The release also encourages users to get involved with the FiftyOne open source community through various channels such as Slack, Meetups, GitHub, and blogs.

This week's FiftyOne tips and tricks blog covers efficient batch edits, parallelizing image loads, excluding samples with certain labels, listing view stages and combining them, and disabling outputs in a Jupyter notebook. The community is encouraged to join the FiftyOne Slack channel for more discussions and assistance.

In this week's FiftyOne Plugin, we built a Concept Traversal plugin that allows users to navigate through the space of concepts using text prompts. This experimental plugin is jointly developed by Jacob and Ibrahim Manjra. The Concept Traversal Panel has three operators: open_traversal_panel, traverse, and get_sample_url. Users can start from an image in their dataset and add text concepts with relative strengths to move around in the multimodal embedding space. This plugin is meant to inspire new approaches to interacting with data.

Voxel51 Inc., a company that enables developers, scientists, and organizations to build high-quality visual AI datasets and computer vision models, is celebrating its fifth anniversary. The open source FiftyOne community has seen tremendous growth over the past year, with increases in GitHub stars, Computer Vision Meetup members, Slack members, repositories depending on FiftyOne, and open source contributors. Voxel51 sponsors a network of 13 Computer Vision Meetups around the world and has also become a proud sponsor of the AI, Machine Learning, and Data Science Meetup network with 12 groups globally. The company released 14 versions of FiftyOne Teams over the past 12 months, introducing new features such as dataset versioning, delegated operations, and more.

In a detailed exploration of video annotation techniques, the article outlines the use of FiftyOne's open-source software to manage and label the WLASL American Sign Language dataset from Kaggle, illustrating how modern video datasets can encompass multiple layers of information such as static detections, frame-level labels, temporal events, and video classification outputs. The tutorial emphasizes the importance of scalable video annotation workflows for organizing large volumes of video data, crucial for tasks like action recognition and gesture analysis in computer vision applications. By showcasing various annotation strategies including sample detections, frame detections, and temporal detections, it demonstrates how these can coexist within a single dataset to capture dynamic information. The guide also highlights how such structured annotation strategies can improve model quality and accelerate experimentation in video-based AI applications, suggesting that these can be extended further with FiftyOne's capabilities to support advanced video analytics pipelines.

In week eight of "Ten Weeks of Plugins," a Reverse Image Search Plugin has been developed for FiftyOne, an open-source tool for managing computer vision datasets. This plugin leverages the existing vector search functionality in FiftyOne to enable users to query their dataset with an image that is not part of it and find the closest matches. The plugin includes two operators: open_reverse_image_search_panel and reverse_search_image, allowing users to select a query image from their local filesystem or by URL. It also provides options for specifying the number of results to return and selecting the similarity index to use. This plugin extends FiftyOne's vector search functionality beyond the data in the dataset and offers a template for working with local or remote resources, tailoring it to users' desired workflows.

The October 12th Computer Vision Meetup featured presentations on various topics such as advancing civil engineering inspections with computer vision, the evolving landscape of AI development, and the introduction of DeciDiffusion for text-to-image generation. Attendees also had the opportunity to vote for their favorite charity, The Coalition for Rainforest Nations (CfRN), which received a $200 donation from the event. Upcoming Meetups will cover topics like getting started with FiftyOne and other computer vision tools.

This week's FiftyOne tips and tricks blog focuses on video labels. The blog covers different types of label data in video datasets, such as detections, classification, tracks, timed events, etc., and how to organize them using the open-source machine learning toolset, FiftyOne. It provides examples of adding sample level detections, frame level detections, temporal detections, and classification labels to a video dataset. The blog also highlights other features available in FiftyOne, such as masks, grouping videos, and computing embeddings.

Voxel51 has released general availability updates for FiftyOneOne 0.22.1 and FiftyOne Teams 1.4.2, which include numerous enhancements and fixes to streamline computer vision workflows. These updates provide improvements in the FiftyOne App, FiftyOne Core, and FiftyOne Plugins, as well as additional features for FiftyOne Teams such as user permissions, dataset versioning, cloud-backed media, and enterprise security. The release notes contain a full rundown of additional enhancements and bugfixes in both updates.

The FiftyOne community has recently highlighted Protex AI's use of its platform for monitoring worker health and safety in various settings using computer vision technologies. In September, there were point releases for both open source and Teams versions of FiftyOne, introducing new features such as dataset versioning, native Ultralytics integration, and delegated operations. The community has also grown with over 2,100 members on the FiftyOne Community Slack channel. Voxel51 sponsors various Computer Vision Meetups worldwide to expand knowledge in computer vision and complementary technologies.

Human pose estimation is a crucial aspect of modern computer vision systems, integral to applications such as activity recognition and augmented reality. Central to advanced pose estimation models, like HigherHRNet, are heatmaps which help localize and interpret human body structures. These heatmaps, visualized as color gradients on images, provide insights into model confidence and predictions, allowing practitioners to debug and improve model performance. Tools like FiftyOne facilitate the generation, manipulation, and visualization of these heatmaps, making the process more interpretable and actionable. The SWAHR-HumanPose model exemplifies how heatmaps can be used to create keypoint skeletons efficiently, demonstrating low false positive rates and robustness even in challenging conditions. By transforming heatmaps from mere intermediate outputs into primary visual aids, teams can better understand spatial relationships and keypoint localization, enhancing model optimization and iteration cycles. As pose estimation technology continues to advance, the ability to interpret these outputs will be crucial for improving system accuracy and reliability.

This week's FiftyOne tips and tricks blog delves into the use of heatmaps in computer vision. A heatmap is a visual representation that highlights areas of interest or intensity within an image, often used for tasks like object detection, pose estimation, or saliency mapping. In FiftyOne, heatmaps can be created by pointing to the location of a saved map or creating a map in memory and loading it instead. One real-world application of heatmaps is pose estimation, where they are used to create keypoint skeletons within images. The SWAHR Pose Estimation model from CVPR 2021 uses heatmaps for this purpose. By visualizing heatmaps on images, users can gain a better understanding of the model's focus and decision-making, ultimately enhancing its accuracy and performance.

The FiftyOne Active Learning Plugin is a tool designed to help users label data faster by intelligently selecting the most informative or ambiguous examples for annotation. This plugin can be integrated directly into the user's annotation workflow, allowing them to prioritize and query crucial data points within the FiftyOne App without any coding necessary. The Active Learning Plugin leverages the modAL framework, which allows users to apply various active learning strategies to models from Sci-kit learn. Currently, this plugin only supports Active Labeling for Classification tasks but could be extended to other computer vision tasks in the future.

The October 5th AI, Machine Learning, and Data Science Meetup covered various topics such as Automatic Prompt Optimization with "Gradient Descent" and Beam Search, Glacier Monitoring with Computer Vision Models, and Building Natural Language Applications with txtai. Reid Pryzant discussed the optimization of prompts for large language models (LLMs) using a nonparametric solution inspired by numerical gradient descent. Nora Gourmelon presented her work on extracting glacier front positions from satellite images using deep learning techniques. David Mezzetti introduced txtai, an open-source tool for semantic search and LLM orchestration. The event also included Q&A sessions and provided information about upcoming Meetups and ways to get involved in the Computer Vision community.

Badger is an open-source Python library designed to simplify the creation, management, and customization of GitHub badges directly from the command line. Inspired by the challenges faced during the PapersWithData project, which involved data wrangling and badge creation, Badger addresses these issues by allowing users to create badges from local or remote SVG files, edit and store them, and easily manage configurations without the need for repeated manual code retrieval. It supports operations such as printing badges to Markdown, copying them to the clipboard, listing all badges, and cloning existing badges with variations. Badger also offers a playful feature that uses GPT-4 to generate AI-created SVG badges. The library is lightweight and user-friendly, intended to streamline badge management for all GitHub projects, and encourages users to explore other open-source projects by Voxel51, such as FiftyOne and VoxelGPT, to enhance their machine learning and computer vision workflows.

The International Computer Vision Conference (ICCV) is taking place in Paris from October 4-6, featuring a variety of exciting events and opportunities for attendees. One key attraction at the conference is Voxel51 Booth #55, which offers visitors three main reasons to visit: exploring new ICCV datasets using open source FiftyOne, meeting fellow enthusiasts in the computer vision community, and scoring epic swag from Voxel51. For those unable to attend ICCV, there are other ways to connect with the FiftyOne community, including upcoming events, getting started with FiftyOne, following on GitHub, and joining the Slack community.