September 2026 Summaries

Voxel51 describes using FiftyOne Skills to curate a balanced subset of Hugging Face’s approximately 900 GB LeRobot community dataset, which contains 1,755 contributor datasets spanning 50 robot embodiments. An AI agent scanned metadata, selected up to 10 episodes per embodiment, downloaded only required video and Parquet shards, and created a 497-episode FiftyOne dataset with episode-level metadata and media references rather than copied pixel data. Each episode was embedded with Qwen3-VL-Embedding-2B, enabling similarity search, UMAP visualization, uniqueness and representativeness scoring, and text-to-video retrieval across different robot types. The analysis found that many clips were near-duplicates because the sampled episodes often came from the same recording sessions, while unusually high uniqueness scores frequently identified very short or broken recordings, illustrating the need to combine embedding-based metrics with duration and quality checks. The dataset includes 75 saved views for robot types, tasks, frame rates, and curation goals, and curated subsets can be exported as separate LeRobot v3 repositories by source dataset for policy training.

Voxel51 compared Claude Sonnet 5, Claude Opus 5, GPT-5.1, and Gemini 3.1 Pro by running identical FiftyOne Agent conversations with the same prompts, tools, skills, and dataset, using provider-reported API token usage to measure actual costs. Across three core tasks involving image description, dataset summarization, and creating a dataset filter, GPT-5.1 was the least expensive at $0.109 and Claude Opus 5 the most expensive at $1.700, while Claude Sonnet 5 achieved the highest quality score of 8.67 out of 10 for $0.331. All models passed the task rubric, but their costs, token consumption, and response quality differed substantially, showing that token pricing alone does not predict operational expense. More demanding tasks highlighted failure behavior as a major cost risk: GPT-5.1 generally stopped quickly when unsuccessful, whereas Sonnet and Gemini sometimes repeatedly pursued dead ends, producing single-task costs above $7. The results are limited by single runs, an isolated sandbox dataset, conservative Claude cost estimates due to cache accounting, and a headless test setup that affected some browser-dependent operations, but they suggest GPT-5.1 is the value-oriented choice and Claude Sonnet 5 offers the strongest quality-per-dollar balance.

A Voxel51 field guide compares five LeRobot v3.0 manipulation datasets for vision-language-action training: two real teleoperated datasets, fmb_multi on a Franka Panda and rh20t_cfg1 on a Flexiv arm, and three synthetic datasets, robocasa-MG_100, robotwin_unified, and InternData-A1. It emphasizes that real robot data is costly and slow to collect because it requires operators, hardware, prepared scenes, and safety measures, leaving the real sources with only 1,804 and 4,258 episodes, while InternData-A1 contains 637,498 simulated trajectories. The synthetic datasets represent distinct strategies: MimicGen recombines 1,250 human demonstrations into varied simulated scenes for RoboCasa, RoboTwin uses an LLM to generate task code and a VLM to detect and repair failures without seed demonstrations, and InternData-A1 programmatically composes skills, assets, and scenes without demonstrations. Each dataset provides different robot embodiments, camera streams, state representations, action spaces, licenses, and export sizes, with several FiftyOne releases being subsets of larger upstream collections. InternData-A1 is the only dataset discussed whose paper reports zero-shot sim-to-real transfer after synthetic-only pretraining, although the guide notes that this evidence is limited and not independently reproduced, while it also cautions that nominal frame rates in the real datasets are inferred rather than derived from per-frame timestamps.

FiftyOne 1.22.0 expands support for Depth Anything V3, enabling monocular depth workflows with confidence and sky masks, reference-view selection, pose and scale alignment, metric-depth indicators, and exports of single images into 3D point clouds viewable within FiftyOne. The post compares the small and base Depth Anything V3 models on six selected Quickstart images using a simple pre-defined near-versus-far depth-ordering test: the small model passed five images, while the base model passed all six but required 1.6 times more processing time per image and 3.9 times more disk space in the reported CPU test. The principal difficult case involved a cat reflected in a mirror, illustrating the known limitations of monocular depth estimation on mirrors, glass, and other surfaces without directly measurable physical depth. It demonstrates that `compute_3d_exports()` can combine a depth estimate and inferred camera information to create a GLB point-cloud scene from one photograph, while emphasizing that teams should evaluate depth models on their own imagery and failure cases before deploying them for applications such as robotics, retail visualization, real estate, or insurance.

LeRobotDataset standardizes robot-learning data using Parquet for states and actions and MP4 for camera streams, addressing fragmentation across robotic platforms while evolving from the per-episode-file v2.x layout to the chunked, streamable v3.0 format. Voxel51’s open-source fiftyone-lerobot-import agent skill automates importing local or Hugging Face LeRobot datasets into FiftyOne by detecting legacy versions, requesting approval before installing dependencies or creating a dataset, converting v2.x sources to v3.0 when needed, and representing each robot episode as a multimodal sample. An end-to-end test using the v2.1 revision of the ALOHA simulated peg-insertion dataset converted all 50 episodes successfully, imported them with no skipped recordings, and enabled episode-level inspection of video, task, duration, state, and action data. The same workflow can also be implemented directly through FiftyOne’s SDK using dataset download, optional LeRobot conversion, and the LeRobotDataset importer, with the lerobot package required only for legacy-format conversion.

FiftyOne 1.22.0 introduces the TII-RATM Drone Racing dataset, a 0.58 GB benchmark containing six indoor quadrotor racing flights recorded as native MCAP episodes with onboard visual-inertial odometry (VIO), motion-capture ground truth, fisheye video, IMU data, and feature tracks. Comparing the drone’s self-estimated position with ground truth shows that four of six flights end at their maximum recorded tracking error, suggesting their VIO drift did not recover before landing, while one ellipse flight reduced a 2.35-meter peak error to 0.67 meters by its final pose. The article proposes the final-error-to-maximum-error ratio as a complement to RMSE, since RMSE can obscure whether localization errors recover or persist; for example, the recovering flight had a lower overall RMSE than several flights that ended at their worst drift. The worst-performing flight, a lemniscate run with 1.66-meter RMSE, can be examined in FiftyOne’s multimodal viewer, which directly displays synchronized 3D trajectories, camera imagery, and tracked features from MCAP topics. A separate analysis finds that the fisheye feature tracker produced usable matches for only about 54% to 58% of camera frames across all flights, highlighting a consistent limitation likely associated with fast indoor motion, blur, and lens distortion. The evaluation approach is presented as applicable to robots, inspection drones, and other vision-based localization systems that can be compared against an independent reference such as motion capture, RTK GPS, or surveyed markers.

A Voxel51 field guide examines five humanoid and bimanual manipulation datasets available in LeRobot v3.0 and FiftyOne, emphasizing “embodiment heterogeneity,” or the incompatibility created by robots’ differing kinematics, state and action spaces, camera configurations, and sensing modalities. The datasets span real-world AgiBot G2, Unitree G1, and Dexmate Vega-1 systems alongside simulated GR1T2 and Reachy2 humanoid torsos, with state vectors ranging from 29 dimensions for the Unitree G1 to 216 for AgiBotWorld2026, partly due to added reinforcement-learning collection signals. The paired TAVIS simulation datasets provide a controlled example, using identical tasks, cameras, and teleoperation procedures but differing in state size because the two robots have different degrees of freedom. HIW-500 records household teleoperation with a legged humanoid, AgiBotWorld includes human-intervened RL rollouts, and T-Rex adds extensive fingertip tactile sensing for dexterous bimanual manipulation. Because none share a unified state or action representation, they cannot be directly combined for cross-embodiment training without alignment approaches such as normalizing actions to shared end-effector representations, while licenses range from unrestricted MIT to non-commercial CC-BY-NC-SA.

A Voxel51 engineer describes rebuilding a poorly documented 2024 human-robot gesture classifier using FiftyOne to investigate why it performed near chance and to improve sim-to-real transfer. Using RoCoG-v2 synthetic and real video data, YOLO pose estimation, skeleton-based feature normalization, unified temporal gesture labels, and causal sliding-window classifiers, the project found that data preparation and representation choices mattered far more than switching between LSTM and temporal convolutional architectures. Normalizing poses, using two-second windows, correcting inconsistent labeling conventions, and adding only 204 real clips substantially improved real-world results, with real data outperforming a much larger synthetic set on real test footage. The best model trained on synthetic plus real data reached 81–82% accuracy on the RoCoG real test set and 51.3% zero-shot accuracy on 840 previously unseen clips recorded from the author’s robot, more than doubling the earlier 24% result. Results also showed strong variation by gesture class, with several gestures recognized nearly perfectly while others remained at or below chance, illustrating that aggregate accuracy can obscure important limitations.

FiftyOne 1.22.0 adds the egocentric-emg-force dataset to its Dataset Zoo, containing eight MCAP-recorded household-task episodes that combine chest-mounted RGB-D video, wrist IMU data, hand skeletons, eight-channel forearm EMG, and vision-derived per-finger force estimates. The article evaluates whether processed EMG envelopes predict these force estimates by rectifying, averaging, resampling, and smoothing EMG signals before performing per-hand, per-episode lag-search cross-correlations. Across 16 hand-episode pairs, the strongest observed correlation was 0.426, 13 pairs were below 0.3, and optimal lags varied from -230 to 200 milliseconds rather than consistently matching the dataset card’s documented 80-millisecond EMG lead. These results suggest that the force channel should be regarded as an independent estimate derived from pose and depth, rather than a direct substitute for physical force measurements or raw muscle activity. The post also describes how FiftyOne can query episode metadata and inspect multimodal streams, and presents the lag-search workflow as a reusable method for evaluating alignment and agreement between independently sampled MCAP sensor topics.

A Voxel51 field guide presents 11 robot-learning datasets reformatted into LeRobot v3.0 and inspectable through FiftyOne, spanning real and simulated data from Franka arms, bimanual systems, humanoids, and tactile-sensing platforms. It argues that synchronized episodic alignment of visual observations, robot states, actions, and language is more important for vision-language-action training than model architecture alone, since misaligned streams undermine action grounding. LeRobot v3.0 is positioned as a compressed, training-oriented format for discrete synchronized episodes, contrasting with MCAP’s lossless continuous sensor logs for applications such as SLAM and sensor fusion, while FiftyOne represents each episode as a multimodal sample whose detailed streams are accessed through media references. The collection includes datasets for humanoid household teleoperation, contact-rich assembly and tactile manipulation, multi-camera real-world demonstrations, and synthetic data generated through simulation, augmentation, or automated task programming; four exports contain their complete sources, while seven are constrained subsets of much larger repositories. The guide also documents variations in robot embodiments, sensor configurations, task labels, licensing, and importer limitations, notably that the FiftyOne DROID export supports RGB streams but not its upstream depth-video and point-cloud data.

Voxel51’s segment-level embeddings are designed for multimodal physical AI recordings in which important behaviors may occupy only brief intervals of much longer robot demonstrations, driving logs, or sensor streams. Unlike episode-level embeddings, which compress an entire recording into one vector and can obscure short events, segment-level embeddings represent smaller time windows and return precise matching moments for similarity or natural-language searches. The system can be configured by embedding model, sensor stream, segment duration, and visualization settings, enabling users to focus on relevant views and choose granularity appropriate for fine actions or longer behaviors. By linking each embedding to its episode, stream, and time range, Voxel51 supports workflows such as behavioral search, failure analysis, and dataset curation, allowing teams to locate events like gripper slips or pedestrian crossings directly within recordings while retaining the option to embed data at episode, image, object, point-cloud, or segment levels.

FiftyOne is presented as a data-flywheel platform for visualizing, curating, annotating, and evaluating multimodal physical AI datasets, including images, video, 3D point clouds, sensor streams, and robotics episodes. Its open-source tools support per-sample annotation for classifications, bounding boxes, segmentation masks, polylines, 3D cuboids, and custom label attributes, while users can define schemas with a field name, label type, and classes before annotating. The walkthrough uses 27,558 malaria-cell images to demonstrate image-level infection classification and parasite segmentation, using manual brushes and polygons as well as click-to-segment assistance from SAM 2, which can be refined with positive and negative points. FiftyOne also supports custom plugins, such as tools for editing vision-language-model captions associated with robot episodes. For large-scale annotation, Voxel51’s enterprise offering adds collaborative workflows, annotation workforces, GPU-backed larger models, and auto-labeling models or agents, aiming to help teams prioritize important subsets, identify model failures, and iteratively improve training data without leaving the platform.

Defense and aerospace visual AI teams face the same long-tail data challenges as other robotics sectors, including identifying rare safety-critical cases, mining model failures, and scaling analysis across growing sensor collections, but classification rules, export controls, and program boundaries prevent them from sharing data, benchmarks, and operational lessons. Because new data collection is costly, difficult to schedule, and often delayed by security review, the article argues that teams cannot solve these gaps simply through more imagery or external benchmarking. An evaluation on the public Aerial Floating Objects search-and-rescue dataset illustrates the issue: a zero-shot Grounding DINO detector detected only about 1.6% of labeled humans and 0.2% of wind/sup-board instances, showing how aggregate performance can conceal severe failures on small, rare, high-value targets. The proposed primary remedy is curation of data already within secure environments through embedding-based exploration, similarity search, and per-class or per-slice evaluation, supported by fully self-hosted, offline, auditable tools that avoid cloud dependencies and vendor lock-in.

Motion-smoothness metrics can help prioritize robot-data recordings for review but cannot determine whether a robot completed its task, as shown by an analysis of 127 simulated RoboLab-EgoX cube-placement episodes. Using FiftyOne’s open-source demo_quality_scorer plugin, the author measured joint and end-effector motion with metrics including spectral arc length, log dimensionless jerk, jerk RMS, and frequency-power ratios, finding that successful episodes were generally smoother than failures overall. However, one failed rollout, RubiksCubeTask_cape_hill_sf0, ranked eighth smoothest, triggered no metric warnings, and was not flagged as a statistical outlier, demonstrating that a fluid trajectory can still place an object incorrectly. The analysis used manually configured two-second windows because the approximately 5.4-second episodes were too short for the plugin’s standard windowing approach, and it emphasizes that scores are relative to the evaluated batch rather than universal quality measures. The article concludes that motion and sensor-quality scoring is valuable for triage, particularly for identifying issues such as sensor dropouts or rough execution, but should supplement rather than replace task-success labels and human inspection.

Voxel51’s Gemini Vision plugin integrates Google Gemini Omni’s video generation and video-understanding capabilities into FiftyOne, while bundled Markdown-based Agent skills instruct the FiftyOne Agent on when, how, and under what cost and safety constraints to use its operators. The plugin supports text-, image-, reference-, edit-, and extension-based video generation, stores generated clips as dataset samples with provenance metadata, and preserves Gemini interaction IDs so generated videos can be iteratively edited, though imported footage cannot be edited this way. For understanding tasks, Gemini returns structured timestamped events that FiftyOne stores as TemporalDetections, making results seekable, filterable, and usable as dataset labels rather than unstructured chat responses. Skills are discovered automatically through a plugin manifest when enabled, can require user approval before billable operations, and guide the agent in selecting tasks, setting parameters, tagging generated media as synthetic, and reporting failures. In a 30-minute repetitive-video test, static sampling at 0.5 frames per second found a target accurately at about one ninth the token cost of high-thinking agentic analysis, while low-thinking agentic mode missed it, illustrating that agentic processing is not always the most efficient option.

Aria Digital Twin, a Meta Reality Labs research dataset of egocentric recordings in digitized indoor environments, provides Project Aria VRS files and separate ground-truth downloads containing synchronized imagery, annotations, device poses, eye gaze, point clouds, depth, segmentation, synthetic renders, and IMU data. The walkthrough explains how to download one licensed episode, arrange its files for Meta’s data provider, convert its modalities into 12 Foxglove-schema channels in a single MCAP container, reorder messages chronologically for smooth playback, and load the result directly as a multimodal sample in FiftyOne. Key technical considerations include converting ADT’s Y-up coordinates for Z-up viewers, rotating images and 2D boxes consistently, republishing point clouds so they remain visible during playback, retaining static 3D boxes only once to reduce file size and rendering work, and validating geometry beyond basic integrity checks. For a 145-second episode, the process combines roughly 12.4 GB of source files into a 2.5 GB MCAP containing 181,444 synchronized messages, while emphasizing that Meta’s non-commercial license prohibits redistribution or public display of the original data and derived MCAP files.

Manually authoring MCAP files from raw sensor data can produce structurally valid files that nevertheless fail or contain subtle errors during playback, largely because serialization and validation often cannot verify semantic correctness. Common issues include declaring Foxglove schemas with JSON rather than their required protobuf encoding, incorrectly placing full epoch nanoseconds or negative values in the bounded nsec field, using inconsistent calibration field casing or SDK versions, and relying on fragile custom PLY or PCD parsers that silently misread real-world layouts. The most difficult errors involve incorrect coordinate conventions, incompatible calibration matrices, and sensors sharing a schema while using different clock domains, since these files may decode and play normally despite representing wrong geometry or timestamps. Recommended practices include retrieving official schemas and encodings from the SDK, centralizing timestamp conversion, inspecting actual message-class documentation, using established file-format libraries, adding numeric assertions for transforms and calibration, and checking playback timelines as well as file structure.

PIVOT, a 56GB drone photogrammetry dataset designed for NeRF and 3D Gaussian splatting research, can also be treated as a robotics-log dataset because each trajectory records a moving camera, dual pose estimates, and a shared static sparse point cloud. The workflow described converts its 103 trajectories across five scenes into individual MCAP episodes for synchronized 3D playback in FiftyOne, while reducing the source download to roughly 27GB by excluding disposable COLMAP intermediate files. Each episode stores original JPEG images, repeated camera calibration, measured drone poses, available COLMAP-optimized poses, and a scene point cloud, using coordinate transformations to reconcile PIVOT’s NED/OpenGL conventions with the Z-up OpenCV/ROS conventions expected by Foxglove and FiftyOne. Since original image timestamps are unavailable, frames receive a synthetic 10 fps clock, and absent COLMAP poses are skipped rather than inferred. The resulting multimodal dataset supports inspection of flight paths and pose disagreement in a shared 3D environment, though users are advised to validate rendered episodes before batch conversion and account for wide-angle fisheye calibration issues when processing all trajectories.

Voxel51 has added native support for Hugging Face’s LeRobot episode-based robotics datasets, enabling teams to visualize, search, and curate synchronized camera feeds, robot states, actions, sensor signals, annotations, and predictions within a shared timeline. As LeRobot’s ecosystem expands, with nearly 200 datasets and a large community collection spanning many robot types, Voxel51 positions its platform as a way to identify represented behaviors, failures, redundant demonstrations, and coverage gaps before data is used for training. The commercial Voxel51 platform provides natural-language, metadata, label, visual-similarity, and embedding-based search across episodes and fine-grained temporal segments, allowing users to retrieve specific behaviors or failure moments and save them as reusable training, evaluation, or annotation cohorts. Voxel51 also supports MCAP, a time-series format commonly used for ROS-based sensor recordings, through the same visualization and curation workflows, while open-source FiftyOne offers native LeRobot visualization and temporal tagging but not Voxel51’s enterprise search and embedding capabilities.