Self-Adapting Recurrent Models for Object Pushing from Learning in Simulation

Abstract

Planar pushing remains a challenging research topic, where building the dynamic model of the interaction is the core issue. Even an accurate analytical dynamic model is inherently unstable because physics parameters such as inertia and friction can only be approximated. Data-driven models usually rely on large amounts of training data, but data collection is time consuming when working with real robots.In this paper, we collect all training data in a physics simulator and build an LSTM-based model to fit the pushing dynamics. Domain Randomization is applied to capture the pushing trajectories of a generalized class of objects. When executed on the real robot, the trained recursive model adapts to the tracked object's real dynamics within a few steps. We propose the algorithm Recurrent Model Predictive Path Integral (RMPPI) as a variation of the original MPPI approach, employing state-dependent recurrent models. As a comparison, we also train a Deep Deterministic Policy Gradient (DDPG) network as a model-free baseline, which is also used as the action generator in the data collection phase. During policy training, Hindsight Experience Replay is used to improve exploration efficiency. Pushing experiments on our UR5 platform demonstrate the model's adaptability and the effectiveness of the proposed framework.

Reinforcement Learning With Vision-Proprioception Model for Robot Planar Pushing

“…Under human interference, the robot can consistently switch the pushing side and keep pushing the object to the target. We use the same online controller (Ruppel et al, 2018 ) to translate Cartesian motions of the pusher into joint-space robot commands as in our previous pushing research (Cong et al, 2020 ).…”

“…Solutions to most robot manipulation problems can be divided into model-based or model-free methods. Our previous work (Cong et al, 2020 ) focused on a model-based method; we built a data-driven recurrent model which adapts to the real interaction dynamics after several pushing interactions using the proposed RMPPI algorithm as the controller. However, one limitation of the previous method is that the robot cannot effectively switch pushing sides according to the object's current pose during the pushing process.…”

A Survey of Meta-Reinforcement Learning

“…However, most methods train a meta-RL agent in simulation instead of on physical robots, where different tasks can be easily created by changing the simulator parameters, and then deploy the meta-trained robot in the real world. This process, known as sim-to-real transfer [272], significantly reduces the meta-training cost and is adopted by most methods that apply meta-RL to robotics [265,2,11,37,106,200,202,215,264,73,111,113,86]. Sim-to-real transfer is depicted in Figure 13.…”

“…( 2) Adapting the dynamics model may be much easier than adapting the control policy in some cases, such as in tasks where task difference is defined by various dynamics parameters. Similar to the model-free case, both task-inference methods [156,37,18,273] and PPG methods [156,106,10] have been considered to adapt the dynamics model for model-based control (there is no black-box model-based method, as learning the dynamic models is just a common way to do task inference).…”

Maximizing the Use of Environmental Constraints: A Pushing-Based Hybrid Position/Force Assembly Skill for Contact-Rich Tasks

“…In future work, the aforementioned results will be combined with other learning-based pushing methods [31] (our previous work) to improve the efficiency and safety of the tending task.…”