Fast Model Identification via Physics Engines for Data-Efficient Policy Search

Abstract

This paper presents a method for identifying mechanical parameters of robots or objects, such as their mass and friction coefficients. Key features are the use of off-the-shelf physics engines and the adaptation of a Bayesian optimization technique towards minimizing the number of real-world experiments needed for model-based reinforcement learning. The proposed framework reproduces in a physics engine experiments performed on a real robot and optimizes the model's mechanical parameters so as to match real-world trajectories. The optimized model is then used for learning a policy in simulation, before real-world deployment. It is well understood, however, that it is hard to exactly reproduce real trajectories in simulation. Moreover, a near-optimal policy can be frequently found with an imperfect model. Therefore, this work proposes a strategy for identifying a model that is just good enough to approximate the value of a locally optimal policy with a certain confidence, instead of wasting effort on identifying the most accurate model. Evaluations, performed both in simulation and on a real robotic manipulation task, indicate that the proposed strategy results in an overall timeefficient, integrated model identification and learning solution, which significantly improves the dataefficiency of existing policy search algorithms.

Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience

“…The closest to our approach are the methods from [30,31,32,33,34] that propose to iteratively learn simulation parameters and train policies. In [30], an iterative system identification framework is used to optimize trajectories of a bipedal robot in simulation and calibrate the simulation parameters by minimizing the discrepancy between the real world and simulated execution of the trajectories.…”

Sim-to-(Multi)-Real: Transfer of Low-Level Robust Control Policies to Multiple Quadrotors

“…For example, Christiano et al [9] learn inverse dynamics models from data gradually collected from a real robotics system, while transferring trajectory planning policy from a simulator. Chebotar et al [10] and Zhu et al [11] transfer manipulation policies by iteratively collecting data on the real system and updating a distribution of dynamics parameters for the simulator physics engine. Similar principles work for the problem of humanoid balancing [12].…”

Sim-to-Real: Learning Agile Locomotion For Quadruped Robots

“…Reality gap is the major obstacle to applying deep RL in robotics. Neunert et al [31] analyzed potential causes of the reality gap, some of which can be solved by system identification [32]. Li et al [33] showed that the transferability of open loop locomotion could be increased if carefully measured physical parameters were used in simulation.…”