Learning Robot Control from Simulated Experience
Training control policies in high-throughput differentiable simulators and deploying them zero-shot sim-to-real.
This research theme explores how robots can acquire agile control behaviors from massive amounts of simulated experience. Such behaviors would be slow, expensive, or dangerous to learn through real-world trial and error.
Real-world experience is a scarce resource to gather for robot learning: crashes are costly, run time is limited, and agile maneuvers put hardware at risk. Simulation removes these constraints, at the expense of observation fidelity. However, tasks like whole-body navigation may not require granular detail in sensor observations to extract relevant signals to learn desired behaviors.
Our work trains control policies end-to-end in high-throughput differentiable simulators, where millions of trials and the simulator’s full state are used as training signals. We exploit this transparency through privileged information, e.g. time-of-arrival maps for navigation and intruder states for interception. This information shapes the training loss but is never available at inference, so the deployed policies rely only on onboard sensing.
Differentiable quadrotor dynamics let us compute analytical policy gradients directly through the simulation, yielding better sample efficiency and smoother control than model-free alternatives like PPO. We have observed this trend in diverse applications: static goal navigation, dynamic target tracking, and collision avoidance.
Crucially, the learned behaviors survive the transfer out of simulation: our navigation policy has flown hundreds of meters through forests and obstacle arenas, day and night, at speeds up to 4 m/s, using only what the robot can sense onboard.
