The Human Corrected the Policy Without Running the Robot

Interactive robot post-training usually means repeatedly running the policy on hardware and waiting for human intervention. HIL-UMI instead queries the current policy on the observation stream from a handheld Universal Manipulation Interface without executing the predicted motion. An energy score targets collection where human and policy trajectories disagree, while low advantage predictions identify segments that refine a progress estimator. Across four physical tasks, the policy improved over supervised fine-tuning and beat HG-DAgger on table cleanup with lower per-frame collection time.