Residual Reinforcement Learning for Robot Control

Johannink, Tobias; Bahl, Shikhar; Nair, Ashvin; Luo, Jie; Kumar, Avinash; Loskyll, Matthias; Ojea, Juan Aparicio; Solowjow, Eugen; Levine, Sergey

doi:10.48550/arxiv.1812.03201

Cited by 12 publications

(24 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A general avenue to addressing the sample complexity in RL is the deliberate use of inductive bias or prior knowledge to aid the exploratory process. This includes reward-shaping [5], [6], [7], curriculum learning [8], learning from demonstrations [9], [10], and the use of behavioural priors [11], [12], [13], [14]. The incorporation of prior knowledge in the form of behavioural priors has been gaining increasing traction in recent years.…”

Section: Learned Controllers Classical Controllersmentioning

confidence: 99%

“…RLBP approaches can directly query an action from the prior at any given state. This allows for a diverse range of mechanisms for introducing inductive bias during training, including regularisation [15], [16], [17], exploration bias [12], [13], [17] and residual learning [11], [18].…”

Section: Learned Controllers Classical Controllersmentioning

confidence: 99%

“…The residual reinforcement learning framework [11], [18], [38] focuses on learning a corrective residual policy for a control prior. The executed action a t is generated by summing the outputs from a control prior and a learned policy, i.e., a t = ψ(s t ) + π θ (s t ).…”

Section: A Residual Reinforcement Learningmentioning

confidence: 99%

“…A. Baselines 1) Residual Reinforcement Learning: Implementation of the residual reinforcement learning algorithm proposed by [11]. 2) KL Regularised RL: Modified SAC algorithm which utilises a KL regularised objective towards a prior behaviour as proposed by [41].…”

Section: Evaluation Of Training Performancementioning

confidence: 99%

See 3 more Smart Citations

Bayesian Controller Fusion: Leveraging Control Priors in Deep Reinforcement Learning for Robotics

Rana¹,

Dasagi²,

Haviland³

et al. 2021

Preprint

View full text Add to dashboard Cite

We present Bayesian Controller Fusion (BCF): a hybrid control strategy that combines the strengths of traditional hand-crafted controllers and model-free deep reinforcement learning (RL). BCF thrives in the robotics domain, where reliable but suboptimal control priors exist for many tasks, but RL from scratch remains unsafe and data-inefficient. By fusing uncertainty-aware distributional outputs from each system, BCF arbitrates control between them, exploiting their respective strengths. We study BCF on two real-world robotics tasks involving navigation in a vast and long-horizon environment, and a complex reaching task that involves manipulability maximisation. For both these domains, there exist simple handcrafted controllers that can solve the task at hand in a risk-averse manner but do not necessarily exhibit the optimal solution given limitations in analytical modelling, controller miscalibration and task variation. As exploration is naturally guided by the prior in the early stages of training, BCF accelerates learning, while substantially improving beyond the performance of the control prior, as the policy gains more experience. More importantly, given the risk-aversity of the control prior, BCF ensures safe exploration and deployment, where the control prior naturally dominates the action distribution in states unknown to the policy. We additionally show BCF's applicability to the zeroshot sim-to-real setting and its ability to deal with out-ofdistribution states in the real-world. BCF is a promising approach for combining the complementary strengths of deep RL and traditional robotic control, surpassing what either can achieve independently. The code and supplementary video material are made publicly available at https://krishanrana.github.io/bcf.

show abstract

Section: Learned Controllers Classical Controllersmentioning

confidence: 99%

Section: Learned Controllers Classical Controllersmentioning

confidence: 99%

Section: A Residual Reinforcement Learningmentioning

confidence: 99%

Section: Evaluation Of Training Performancementioning

confidence: 99%

See 2 more Smart Citations

Bayesian Controller Fusion: Leveraging Control Priors in Deep Reinforcement Learning for Robotics

Rana¹,

Dasagi²,

Haviland³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…We are interested in applications to robotics control, which typically have continuous state and action spaces (Collins et al, 2005;Abbeel et al, 2007;Levine et al, 2016). For example, reinforcement learning can be used to learn controllers when the dynamics are unknown (Abbeel et al, 2007;Ross & Bagnell, 2012) or partially unknown (e.g., due to unknown disturbances such as wind or friction) (Akametalu et al, 2014;Berkenkamp et al, 2017;Johannink et al, 2018). Understanding sample complexity is especially important in this application, since the eventual goal is for robots to be able to learn based on real world experience, which can be very costly to obtain.…”

Section: Introductionmentioning

confidence: 99%

Sample Complexity of Estimating the Policy Gradient for Nearly Deterministic Dynamical Systems

Bastani

2019

Preprint

View full text Add to dashboard Cite

Reinforcement learning is a promising approach to learning robot controllers. It has recently been shown that algorithms based on finite-difference estimates of the policy gradient are competitive with algorithms based on the policy gradient theorem. We propose a theoretical framework for understanding this phenomenon. Our key insight is that many dynamical systems (especially those of interest in robot control tasks) are nearly deterministic-i.e., they can be modeled as a deterministic system with a small stochastic perturbation. We show that for such systems, finite-difference estimates of the policy gradient can have substantially lower variance than estimates based on the policy gradient theorem. We interpret these results in the context of counterfactual estimation. Finally, we empirically evaluate our insights in an experiment on the inverted pendulum.

show abstract

Reinforcement Learning on Variable Impedance Controller for High-Precision Robotic Assembly

Luo

Solowjow

Wen

et al. 2019

2019 International Conference on Robotics and Automation (ICRA)

Self Cite

150

View full text Add to dashboard Cite

Precise robotic manipulation skills are desirable in many industrial settings, reinforcement learning (RL) methods hold the promise of acquiring these skills autonomously. In this paper, we explicitly consider incorporating operational space force/torque information into reinforcement learning; this is motivated by humans heuristically mapping perceived forces to control actions, which results in completing high-precision tasks in a fairly easy manner. Our approach combines RL with force/torque information by incorporating a proper operational space force controller; where we also exploit different ablations on processing this information. Moreover, we propose a neural network architecture that generalizes to reasonable variations of the environment. We evaluate our method on the open-source Siemens Robot Learning Challenge, which requires precise and delicate force-controlled behavior to assemble a tight-fit gear wheel set. a) Robot learning for complex assemblies. b) Assembled gear.

show abstract

Residual Reinforcement Learning for Robot Control

Cited by 12 publications

References 0 publications

Bayesian Controller Fusion: Leveraging Control Priors in Deep Reinforcement Learning for Robotics

Bayesian Controller Fusion: Leveraging Control Priors in Deep Reinforcement Learning for Robotics

Sample Complexity of Estimating the Policy Gradient for Nearly Deterministic Dynamical Systems

Reinforcement Learning on Variable Impedance Controller for High-Precision Robotic Assembly

Contact Info

Product

Resources

About