Physical AI refers to AI systems that can operate in and reason about the physical world, combining visual, auditory, spatial, and temporal data to understand what's happening and what actions should follow. Building such systems is challenging due to high-dimensional, unstructured input, lack of clear ground truth, fragmented workflows, and the need for new data infrastructure, scalable annotation workflows, and evaluation pipelines that mimic real-world environments. To overcome these challenges, teams can leverage multimodal annotation tools, model-in-the-loop setups, agentic workflows, and behavioral benchmarks to define success using task-based or behavioral metrics, rather than just classification scores.