How I Trained Action Chunking Transformer (ACT) on SO-101: My Journey, Gotchas, and Lessons

Sherry Chen shares her detailed experience of training the Action Chunking Transformer (ACT) on the SO-101 robot to perform a pick-and-place task, highlighting both successes and challenges. Initially confronted with hardware and data collection issues, including camera disconnections and inadequate training data, Chen refined her approach by standardizing hardware setups, improving data diversity, and enhancing debugging infrastructure. Despite early setbacks, including calibration mismatches and motor failures, her persistence in adjusting training strategies led to significant improvements, achieving a 90% success rate in distribution and 75% out of distribution. This journey underscores the importance of consistent setups, diverse data, and robust debugging tools in robotics training, while also emphasizing the real-world challenges of dealing with hardware limitations and unexpected failures. Chen concludes with plans to expand the task complexity and improve model generalization, offering valuable insights and practical tips for others pursuing similar robotics projects.