Residual Reinforcement Learning for Robot Control

Abstract

Conventional feedback control methods can solve various types of robot control problems very efficiently by capturing the structure with explicit models, such as rigid body equations of motion. However, many control problems in modern manufacturing deal with contacts and friction, which are difficult to capture with first-order physical modeling. Hence, applying control design methodologies to these kinds of problems often results in brittle and inaccurate controllers, which have to be manually tuned for deployment. Reinforcement learning (RL) methods have been demonstrated to be capable of learning continuous robot controllers from interactions with the environment, even for problems that include friction and contacts. In this paper, we study how we can solve difficult control problems in the real world by decomposing them into a part that is solved efficiently by conventional feedback control methods, and the residual which is solved with RL. The final control policy is a superposition of both control signals. We demonstrate our approach by training an agent to successfully perform a real-world block assembly task involving contacts and unstable objects.

TossingBot: Learning to Throw Arbitrary Objects with Residual Physics

“…This approach provides a wider range of data-driven corrections that can compensate for noisy observations as well as dynamics that are not explicitly modeled. These benefits are also observed in concurrent work on residual reinforcement learning [16,30] in block-assembly and object manipulation tasks.…”

A Comparison of Action Spaces for Learning Manipulation Tasks

“…Indeed, it is often necessary in practice to implement gravity compensation to learn successful manipulation strategies using torque control [11], [5]. Johannink et al have shown that formulating the learning task to learn residuals to a hand-crafted model-based controllers can improve learning efficiency [12]. Another approach is to design the action space in terms of references for an underlying model-based controller.…”

“…While there is some work that formulates the action space in terms of a low-level Cartesian position controller for tasks such as block stacking [12], pushing, and pick and place [30], there is surprisingly little work in reinforcement learning that uses impedance control to combine the ideas of task space control with the mechanical compliance necessary to perform delicate manipulation tasks under uncertainty, and no work to our knowledge that attempts a systematic comparison between alternative policy structures in this domain.…”

Differentiable Algorithm Networks for Composable Robot Learning

“…Imperfect models are common in robotic systems. They have been studied widely in various contexts, e.g., to account for uncertainty in the planning model [13,16,23,49,44] or to directly learn strategies that are robust against imperfections in environment models [11,2], policies [24,42], or approximate algorithms [40]. Unlike the earlier work, the DAN commits to the algorithm choices, but adapts the models to compensate for imperfections through end-to-end training from data.…”