Sim-to-Real Gesture Classification for Robots in FiftyOne

A Voxel51 engineer describes rebuilding a poorly documented 2024 human-robot gesture classifier using FiftyOne to investigate why it performed near chance and to improve sim-to-real transfer. Using RoCoG-v2 synthetic and real video data, YOLO pose estimation, skeleton-based feature normalization, unified temporal gesture labels, and causal sliding-window classifiers, the project found that data preparation and representation choices mattered far more than switching between LSTM and temporal convolutional architectures. Normalizing poses, using two-second windows, correcting inconsistent labeling conventions, and adding only 204 real clips substantially improved real-world results, with real data outperforming a much larger synthetic set on real test footage. The best model trained on synthetic plus real data reached 81–82% accuracy on the RoCoG real test set and 51.3% zero-shot accuracy on 840 previously unseen clips recorded from the author’s robot, more than doubling the earlier 24% result. Results also showed strong variation by gesture class, with several gestures recognized nearly perfectly while others remained at or below chance, illustrating that aggregate accuracy can obscure important limitations.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.