July 2026 Summaries

FiftyOne, an open-source toolkit by Voxel51, has introduced a novel plugin that allows users to interact with their computer-vision datasets through plain-English queries, leveraging the capabilities of Laguna S 2.1, Poolside's open-weight coding model. This model, which excels in agentic coding tasks, converts natural language questions into FiftyOne view queries, executing them to provide filtered dataset views without requiring users to write any code. Laguna S 2.1, despite its lack of vision capability, adeptly reasons over dataset schemas and error messages to refine queries, while delegating tasks that necessitate visual perception to a vision-language model. This self-correcting loop, central to its functionality, enables seamless dataset interaction and curation, marking a significant advancement in making complex data analytics more accessible and intuitive for users.

TotalSegmentator is an open-source tool designed for medical image segmentation, capable of labeling over 100 anatomical structures from CT or MR volumes without requiring per-organ training. Built on the nnU-Net framework, it is compatible with multiple operating systems and offers robust performance across varied scanning protocols. When paired with FiftyOne, another open-source tool from Voxel51, users can visually inspect and evaluate model predictions against ground truth data to identify segmentation failures, specifically for small, low-contrast structures like the pancreas and adrenal glands, which are often masked by high aggregate metrics such as mean Dice. FiftyOne enhances the workflow by allowing users to sort and filter data to uncover model shortcomings and labeling errors, transforming the evaluation process into a comprehensive data audit. Both tools aim to provide deeper insights into model performance, helping users address errors and improve data quality efficiently.

The blog post discusses Fish-Vista, a dataset comprising 60,000 images of museum fish specimens across 1,900 species, utilized to explore and test the capabilities of the FiftyOne tool, particularly in segmentation and taxonomy. Unlike the Community Fish Detector (CFD) used in Part 1, which focused on detecting fish in diverse underwater environments, Fish-Vista allows for species classification, trait identification, and pixel-level segmentation of nine anatomical structures. The study found that a detector trained on wild footage could effectively identify fish in the high-quality museum images without fine-tuning, achieving a mean average precision (mAP) of ~0.97. This cross-domain robustness highlights the potential of using field-trained models for auto-annotation in a different modality. The blog also emphasizes the ability of FiftyOne to work with imbalanced species data and how it facilitates comparative anatomy through trait masks, enabling users to perform complex queries and analyze species morphology efficiently.

Autonomous vehicles generate vast amounts of data, yet identifying rare, safety-critical instances within this data remains a significant challenge due to the inherent imbalance in datasets. The "long tail" of rare objects and scenarios, such as donkeys on roads or pedestrians using mobility aids, often escapes detection because these cases are neither frequent nor anticipated. As datasets grow, the problem shifts from data collection to data discovery, where the focus is on finding these rare examples within existing data. Traditional tools and metrics like label-based search and mean average precision (mAP) often fail to highlight these rare cases, as they are designed for more common categories. Instead, similarity search using image embeddings is proposed as a solution to surface rare cases by analyzing the visual content of images rather than relying solely on labels. This approach allows for the retrieval of rare scenarios without predefined classes, improving the discovery process as models evolve.

Building high-performing AI models is heavily reliant on robust and accurate data, yet transforming raw data into a labeled, model-ready state remains a significant operational challenge. The increasing adoption of auto-labeling aims to alleviate this, but specialized tasks often still demand manual effort, as off-the-shelf models struggle with domain-specific concepts and custom models require extensive data collection. Vision-language models (VLMs) have emerged as an alternative, leveraging pretraining on large image-text pairs to generalize across new categories with minimal labeled data. FiftyOne's Agentic Labeling offers a streamlined, no-code workflow for leveraging VLMs to rapidly experiment and scale high-quality labeling, allowing teams to train reusable labeling agents with natural language prompts and visual samples. This approach not only reduces the data tax associated with custom models but also shifts annotators from manual labeling to refining labeled baselines, thus enhancing efficiency and focusing expertise where it matters most. As AI models mature, the emphasis is shifting from data volume to coverage and quality, with FiftyOne's platform enabling seamless integration of curation, model evaluation, and smart data selection to optimize performance and address model failures effectively.

The article provides a detailed exploration of the CropAndWeed dataset, which is designed for automated plant-specific weed intervention. Developed by the AIT Austrian Institute of Technology, this dataset includes 111,953 crop and weed instances across 8,034 field images. The text highlights several data-quality challenges, such as class imbalance and annotation errors, identified using the FiftyOne App without the need for model training. It describes how the dataset, despite appearing clean on paper, contains issues like near-duplicate images and mislabeled instances. The article emphasizes the importance of examining dataset components accurately, as shown by the Vegetation fallback class inflating class imbalance metrics. The author demonstrates a hands-on method to audit and correct labels, exporting a curated subset for potential retraining. Additionally, the text discusses the limitations of whole-image embedding searches in identifying rare classes and the effectiveness of patch-level searches in addressing these challenges. The dataset's lack of an official train/val/test split is noted, with suggestions for building a triage queue and mining rare classes efficiently using foundation models and the FiftyOne Agent.

The article explores the process of running an open-source model, specifically Google's Gemma 4, on personal hardware using FiftyOne Agent, an adaptable tool designed for physical AI data. The narrative details the challenges and solutions encountered when fitting the model into a single RTX 4070 Ti GPU, highlighting the memory constraints and configuration adjustments required. FiftyOne's flexibility allows users to connect any self-hosted model through a standard chat endpoint, effectively treating it like any other hosted model. This capability is particularly advantageous for handling sensitive data that cannot be shared with public APIs, ensuring that the data and model remain within the same secure environment. The article also emphasizes the importance of understanding the actual resource costs involved in self-hosting models, facilitated by FiftyOne's plugin framework, which provides real-time monitoring of model performance and resource usage.

KubriCount is a synthetic benchmark designed to test foundation models in robotics by reframing counting tasks as verifiable prompt-following problems, emphasizing the importance of grounding instructions to avoid errors like a robot grabbing the wrong block. The article outlines a continuous workflow using the FiftyOne tool, which involves curating, annotating, generating, and evaluating synthetic data to identify and rectify model failures before deployment. KubriCount's dataset includes 110,507 synthetic images with over 7.3 million annotated objects, categorized into five explicit counting granularity levels, each with a target and a distractor set, to rigorously evaluate models' capabilities in distinguishing targets under specific prompts. The study highlights the often-overlooked biases and potential failures in foundation models, as illustrated by the NVIDIA LocateAnything-3B model's performance, which struggled with the hardest disambiguation tasks, often defaulting to visually dominant groups rather than the specified target. By using KubriCount, researchers can visualize generalization gaps and specific failure modes, informing future data generation and model training efforts to enhance the reliability and accuracy of open-world robotics applications.

In the realm of robotics, the scarcity of diverse training data poses a significant challenge to developing general-purpose agents, as robots cannot rely on the extensive datasets available to language and vision AI. Unlike web-sourced data, robot data must be generated through physical or simulated interactions, capturing unique sensor readings and proprioceptive feedback, which cannot be easily scraped from existing sources. The key to improving robot generalization lies in diversifying the training datasets to include rare, long-tail, and out-of-distribution cases that robots may encounter in real-world scenarios. Vision-Language-Action (VLA) models, which integrate vision and language inputs to drive robot actions, highlight the critical shift from model-centric to data-centric approaches in AI, emphasizing the need for curated and varied data to enhance performance. Tools like FiftyOne help identify these rare scenarios by using multimodal embeddings, allowing for more efficient data curation and improved model training. This approach underscores the importance of focusing on data diversity over quantity to achieve better generalization in robotics, as demonstrated by recent industry findings.

FiftyOne 1.19 introduces native support for opening MCAP files, enabling seamless integration of camera, LiDAR, GPS, IMU, and log data into a unified playback environment without requiring a ROS installation. The MCAP format, which serves as a self-describing log container, allows long-term decodability of data by embedding each message's schema within the file. This update allows users to analyze complex datasets like the 10 UAV flights through a Virginia forest, where each flight records data across 31 channels, including stereo cameras and LiDAR, all synced to a single clock. The multimodal viewer in FiftyOne offers a comprehensive interface with six tile types that synchronize playback across various data streams, providing a cohesive exploration of the data. This eliminates the need for separate tools, such as rosbag or Foxglove, to interpret and visualize diverse, timestamped message streams, thereby streamlining the process of turning raw robotic data into usable insights.

MiniMax has introduced a new plugin for FiftyOne that integrates the MiniMax-M3 model, a comprehensive multimodal AI model, into computer vision workflows without requiring fine-tuning or hosting a model server. MiniMax-M3, characterized by its 428-billion parameter architecture and a 1 million token context window, utilizes a novel sparse-attention mechanism to enhance performance. The plugin allows users to execute tasks such as detection, keypoint annotation, classification, and more, by transforming model outputs into structured labels suitable for FiftyOne's labeling pipelines. The model is driven by prompt engineering, providing outputs in JSON format for direct conversion into FiftyOne label objects. This integration facilitates efficient and scalable data annotation processes, with options for bootstrap labeling, semantic search, and event detection across images and video samples. Additionally, the plugin supports three thinking modes—disabled, adaptive, and enabled—allowing customization of the reasoning process based on task requirements.

MOTOR, a dataset created by researchers at IIIT Hyderabad's CVIT lab, addresses the gap in driving datasets by focusing on motorcycles and scooters, which dominate transportation in many regions like India and the Global South. Unlike traditional datasets centered on four-wheeled vehicles, MOTOR encompasses 1,629 annotated maneuver sequences from 16 riders, captured over four weeks of real Indian traffic. This dataset includes synchronized views from front, helmet, rear, and eye-tracker cameras, along with GPS routes, gyroscope data, and rider gaze heatmaps. A subset of this dataset, comprising 324 clips, is available in a grouped-dataset format on Hugging Face, specifically designed for easy exploration using FiftyOne. The dataset supports multimodal analysis by integrating video, gaze, and telemetry data, which is crucial for developing advanced driver-assistance systems for two-wheelers and enhancing safety research. The FiftyOne platform facilitates a comprehensive view of the dataset, although it has some constraints, such as handling GeoLocation fields. The MOTOR dataset is significant for understanding rider behavior, legality, and attention modeling, providing a valuable resource for research and development in autonomous systems and safety applications.

Google's Antigravity CLI is an autonomous coding agent designed to execute complex engineering tasks, such as importing and curating large datasets, by leveraging skills, plugins, and MCP servers to provide domain expertise. In a demonstration, it successfully processed 81,444 images from the WikiArt dataset using FiftyOne Skills, a library of 18 open-source skills with over 80 visual AI operators, completing the task in approximately 67 minutes. The CLI can install its tools, run extensive jobs, fix its bugs, and switch models mid-task without losing progress, as demonstrated when it switched from Gemini 3.5 Flash to Claude Sonnet 4.6 upon hitting a quota limit. The process includes importing images, calculating embeddings, and generating uniqueness scores and UMAP visualizations, providing insights into the dataset's completeness and reliability. Antigravity's agent can adapt plans and scripts autonomously to optimize performance, as seen when it improved import and curation speeds by rewriting scripts and switching processing methods.

FiftyOne introduces a comprehensive solution for managing multimodal time-series data in physical AI, addressing the challenges faced by teams developing autonomous systems, such as data curation bottlenecks and black box debugging. The platform supports synchronized playback of camera, LiDAR, and sensor streams, enabling users to search, visualize, and annotate data in a unified environment. This integration allows for more efficient identification of failure modes and data gaps, facilitating improved model performance through better generalization and reduced data fragmentation. By utilizing FiftyOne, teams can streamline their workflows, reduce the risk of failure, and accelerate the validation and deployment of models by transforming their physical AI stack into a continuous data flywheel.

The FiftyOne Agent, developed by Adonai Vera at Voxel51, emphasizes that an agent's effectiveness is determined more by the tools, skills, and context provided to it than by the model itself. The agent is structured around skills—human-authored workflows that guide the agent—and the Model Context Protocol (MCP), which connects the agent to real operations with over 80 operators for dataset management, model inference, and more. This setup allows the agent to perform practical tasks such as opening datasets, running brain methods, and evaluating models. The agent supports over 100 model providers, enabling flexibility and avoiding vendor lock-in. A critical safety feature is a guardrail check that ensures every tool call is safe, requiring confirmation for potentially risky actions. The FiftyOne Agent also retains conversation history, allowing users to seamlessly continue their workflow. This approach parallels successful tools in coding, focusing on the breadth and organization of tools rather than the underlying model's intelligence. Feedback from users is actively sought to continually improve and expand the agent's capabilities.

FiftyOne Search is a dataset search and exploration tool for physical AI teams working with large, temporal, multimodal recordings containing data such as camera feeds, LiDAR, radar, trajectories, and sensor signals. It creates persistent dataset indexes and embeddings that allow users to search across recordings through natural-language descriptions, visual similarity, derived time-series events, and metadata filters, with these methods usable in combination. Segment-level embeddings enable retrieval of brief relevant moments within long recordings, such as failed robot grasps or unusual driving interactions, rather than treating entire episodes as a single unit. Search results open in synchronized multimodal visualizations that connect camera, 3D, signal, annotation, and prediction data on a shared timeline, helping teams investigate model failures and behavior. The platform also supports dataset curation by identifying recurring failures, coverage gaps, and underrepresented scenarios, allowing teams to improve training and evaluation data and prioritize future collection efforts, including for formats such as MCAP.

The article explores the diverse landscape of agriculture-focused computer vision datasets, emphasizing their critical role in advancing agricultural technology and robotics. It presents a comprehensive guide to 14 open datasets, categorized by tasks such as crop and weed segmentation, plant phenotyping, plant disease classification, and 3D point cloud applications. These datasets leverage various sensor modalities like RGB images, multispectral images, satellite imagery, and 3D point clouds, catering to the unique challenges of agricultural environments where traditional vision models often falter. The article highlights the significance of these datasets in supporting precision agriculture and robotics applications, where understanding plant growth stages, disease identification, and field boundary detection are crucial. Additionally, it underscores the importance of licensing considerations, as some datasets are restricted to non-commercial use, and provides insights into the potential of datasets like LettuceMOTS for developing agricultural robots capable of multi-object tracking.

OctoSense is a comprehensive multimodal robotics dataset featuring hundreds of synchronized driving sequences captured from various platforms such as cars, quadrupeds, and marine vehicles, containing data from stereo RGB and event cameras, infrared, 64-beam LiDAR, IMU, GPS, and CAN-bus, all synchronized to a single clock. It provides ground truth for depth, semantic segmentation, optical flow, odometry, and scene captions, making it a robust tool for testing data handling capabilities. Combined with FiftyOne, an open-source toolkit for visual dataset curation, OctoSense becomes a queryable and browsable dataset that allows for the effective quality assurance of labels, clustering of scenes, and the curation of slices across sequences. FiftyOne enhances the dataset's usability by offering a natural-language visual search and caption embedding search, enabling users to explore and filter data efficiently, making the process of dataset curation more interactive and less reliant on scripting.

KITScenes-LongTail is a curated dataset focusing on rare driving scenarios that pose significant challenges for autonomous driving systems. Comprising 103 real nine-second events, the dataset is integrated into the FiftyOne platform, offering six-camera video captures, a VGGT-Omega 3D reconstruction, and expert reasoning traces in English, Spanish, and Chinese. These scenarios highlight the "long-tail" problem, where autonomous models often lack exposure to infrequent but critical events like foggy conditions or complex maneuvers. The dataset emphasizes the importance of including rare occurrences that standard models typically overlook, aiming to enhance model robustness and safety. It provides a rich resource for benchmarking model performance using multi-maneuver scores and exploring multilingual reasoning, with pre-computed Qwen3-VL embeddings for advanced analytics. The dataset, released under CC BY-NC 4.0, is not for commercial use and represents a significant step towards addressing the statistical challenges in autonomous driving data collection.

Jolia, developed by Raidium, is a 3D CT vision-language foundation model that generates a global [CLS] embedding and 102 named per-organ embeddings from paired chest and abdominal CT scans, without requiring segmentation masks or spatial supervision. FiftyOne, an open-source tool by Voxel51, facilitates the exploration and visualization of these embeddings, allowing for interactive UMAP visualization, per-organ similarity search, and attention heatmap overlays. By using the CT-RATE dataset, the integration of Jolia and FiftyOne demonstrates how per-organ embeddings provide independent views of scans, offering insights into model behavior and interpretability. Jolia's innovative training method, ConQuer, aligns anatomical regions with corresponding report sections, enabling it to achieve state-of-the-art performance in findings classification and report generation. While Jolia and FiftyOne are not intended for clinical diagnosis, they serve as valuable research tools for understanding and improving medical imaging models.

KITScenes Multimodal, developed by the Institute of Measurement and Control Systems at the Karlsruhe Institute of Technology and FZI Research Center for Information Technology, is an extensive European urban driving dataset featuring a dense sensor suite with nine synchronized cameras, seven lidars, and three radars. This dataset aims to enhance the fidelity of autonomous driving datasets by offering high-resolution imagery, detailed lidar point clouds, and comprehensive HD maps annotated in the Lanelet2 format, which provide full topological connectivity. Integrated with the FiftyOne platform, KITScenes Multimodal organizes multiple sensor inputs into single, synchronized samples for more efficient data processing and analysis, making it a valuable tool for developing and testing autonomous driving technologies. The current FiftyOne build serves as a preview with validation scenes from Frankfurt, allowing users to explore the dataset's capabilities and develop data pipelines, though it lacks certain dynamic annotations present in the full corpus.

FiftyOne Enterprise 2.21.0 enhances its data annotation platform with new features like Annotation Workflows, video annotation, and temporal tags to improve team collaboration and streamline the labeling process. The update builds on the open-source FiftyOne 1.18.0 release, adding Enterprise-exclusive features such as multi-stage annotate-and-review pipelines, role-aware task queues, and in-app annotation schemas for consistent labeling. Video annotation is now generally available, allowing users to label object tracks directly in the app with keyframes and interpolation, while temporal tags enable time-based labeling for events. The release also includes quality-of-life improvements like faster app startup, new embeddings, and Transformers 5.x support, making FiftyOne a comprehensive solution for data annotation and management.

The Community Fish Detector is a single-class object detection model designed to identify fish in various aquatic environments, leveraging the comprehensive Community Fish Detection Dataset that contains over 1.9 million images. The dataset, harmonized into a single COCO archive from 17 diverse sources, presents a robust challenge for model evaluation, as it includes environments ranging from murky Danish waters to Australian billabongs. FiftyOne, an open-source tool, enhances this evaluation by providing a visual interface that goes beyond aggregate metrics, allowing users to explore per-source mean Average Precision (mAP) scores and identify domain-specific model weaknesses. It facilitates the detection of annotation errors and unlabeled fish within the dataset, offering a more granular understanding of model performance. This method of evaluation, which involves streaming curated data subsets to avoid downloading the entire dataset, enables users to gain insights into the model’s efficiency across different environments.

CarCrashNet, a collaboration between MIT and the Toyota Research Institute, is an open-source benchmark featuring over 15,000 structural crash simulations, making it the first large-scale dataset of its kind for data-driven crash simulations. This extensive dataset includes simulations for three vehicle models—Dodge Neon, Toyota Yaris, and Chevrolet Silverado—using the open-source solver OpenRadioss and validated against Ansys LS-DYNA and physical crash tests. FiftyOne, an open-source visual AI tool, has been employed to handle the dataset, enabling the exploration and curation of crash videos, static figures, tabular metrics, and learned embeddings. With its ability to create multi-camera group slices and a live benchmark leaderboard, FiftyOne demonstrates its versatility in handling scientific simulation data, even though it was originally designed for visual datasets. While the raw 6.65 TB of per-case field data is not yet publicly available, the demo notebook provides a framework for ingestion, poised for when the data is released post-peer review.

BabyROS is a lightweight alternative to the Robot Operating System (ROS) built on the Zenoh protocol, designed to facilitate simple communication between processes without the complexities of a full ROS installation. It allows data to be published and subscribed to on topics with minimal setup, omitting message semantics to let users define their own schemas. FiftyOne complements BabyROS by transforming the raw data it transports into a structured dataset that can be queried, evaluated, and visualized, enabling real-time curation and error detection in robot perception pipelines. The integration of BabyROS and FiftyOne allows for seamless handling of both 2D and 3D data, supporting live updates and evaluations, and offers a comprehensive toolkit for prototyping and developing AI-driven robotics applications while maintaining flexibility and ease of use.

TwelveLabs and FiftyOne have integrated their technologies to enhance video data handling through natural-language video search, transforming how users interact with video datasets. TwelveLabs provides video foundation models like Marengo and Pegasus, which enable embedding video clips into a shared space with text for easy searching by description, and generating natural-language captions and answers for video content. This integration allows users to embed, search, and caption videos using plain English without the need for local GPUs, as computations occur server-side via the TwelveLabs API. FiftyOne, an open-source toolkit, complements this by offering a visual app for exploring images and video, employing a query language for data slicing, and a "Brain" layer for advanced analysis, thereby facilitating better data curation. This partnership unlocks enhanced functionalities such as natural-language video search, zero-shot captioning, similarity detection, and embedding visualizations, enabling users to manage video datasets more efficiently and intuitively.

TreeMatch, an optimal-transport method for estimating tree density from satellite imagery, is explored through FiftyOne, an open-source tool that provides a deeper understanding of model performance beyond benchmark tables. TreeMatch addresses the challenge of using both strong and weak point annotations, accommodating expert-labeled data alongside noisier pseudolabels, and is benchmarked on the TinyTrees dataset, which includes approximately 11.7 million point-annotated trees across three sensors from Rwanda, China, and France. FiftyOne enhances the evaluation by offering visualization features, such as Keypoints for point annotations, Heatmap overlays for predicted density maps, and sortable fields for per-tile count errors, allowing users to inspect where the model fails and understand label noise. The TinyTrees dataset is available under a CC BY-NC 4.0 license for research and education only, and FiftyOne allows the exploration of TreeMatch using official data loaders and pretrained checkpoints without modifying the code.