Helping the Spot Robot See with Instance Segmentation

Peter Mitrano, a PhD student at the University of Michigan, outlines strategies for enhancing robotic manipulation through instance segmentation, with a focus on perception in robots like the Spot Robot. Emphasizing the importance of accurate data collection and labeling, Mitrano suggests starting with a small set of images to identify issues and iteratively improve model performance. He highlights the necessity of selecting appropriate training data and adapting camera angles to improve detection, advocating for active learning by incorporating missed detections into the training dataset. Mitrano also stresses the value of creating effective labeling schemes, such as labeling object parts instead of whole objects, to improve manipulation tasks. Additionally, he discusses the challenges of projecting 2D image masks into 3D space using RGBD cameras, suggesting methods like fitting planes to depth images and using CDCPD tracking for more reliable 3D object location and shape estimation.