Multiplicative Controller Fusion: Leveraging Algorithmic Priors for Sample-efficient Reinforcement Learning and Safe Sim-To-Real Transfer

Rana, Krishan; Dasagi, Vibhavari; Talbot, Ben; Milford, Michael; Sünderhauf, Niko

doi:10.1109/iros45743.2020.9341372

Cited by 9 publications

(15 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A general avenue to addressing the sample complexity in RL is the deliberate use of inductive bias or prior knowledge to aid the exploratory process. This includes reward-shaping [5], [6], [7], curriculum learning [8], learning from demonstrations [9], [10], and the use of behavioural priors [11], [12], [13], [14]. The incorporation of prior knowledge in the form of behavioural priors has been gaining increasing traction in recent years.…”

Section: Learned Controllers Classical Controllersmentioning

confidence: 99%

“…RLBP approaches can directly query an action from the prior at any given state. This allows for a diverse range of mechanisms for introducing inductive bias during training, including regularisation [15], [16], [17], exploration bias [12], [13], [17] and residual learning [11], [18].…”

Section: Learned Controllers Classical Controllersmentioning

confidence: 99%

“…An alternative approach is to restrict exploration to be governed by the behavioural prior only until the policy is capable of safely exploring on its own accord. [13] proposed a multiplicative fusion strategy for stochastic policies and behavioural priors. This approach utilises a gating function to initially allow exploratory actions to only be sampled from the behavioural prior and gradually transition control towards the policy in order to exploit its own actions.…”

Section: Exploration Bias Using Behavioural Priorsmentioning

confidence: 99%

“…We study two different task domains (shown in Figure 6) to evaluate the performance of BCF as a viable strategy towards exploiting the strengths of RL and classical control, both during training and real-world deployment. To provide direct comparison with previous work [13], we conduct experiments in the PointGoal Navigation task which was first presented by [52] as well as a complex reaching task that requires manipulability maximisation across the trajectory. For both tasks, we assume the sparse reward, long horizon setting and the presence of existing control priors that can provide some structure towards solving the task.…”

Section: A Tasksmentioning

confidence: 99%

“…3) CORE-RL: Implementation of the TD-error based exploration strategy to balance exploration between a control prior and the policy as proposed by [17]. 4) MCF: Our prior work that leverages a fixed gating function to switch between the control prior and policy over the course of training [13]. 5) SAC: Vanilla SAC algorithm using maximum entropy based Gaussian exploration [31].…”

Section: Evaluation Of Training Performancementioning

confidence: 99%

See 4 more Smart Citations

Bayesian Controller Fusion: Leveraging Control Priors in Deep Reinforcement Learning for Robotics

Rana¹,

Dasagi²,

Haviland³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

We present Bayesian Controller Fusion (BCF): a hybrid control strategy that combines the strengths of traditional hand-crafted controllers and model-free deep reinforcement learning (RL). BCF thrives in the robotics domain, where reliable but suboptimal control priors exist for many tasks, but RL from scratch remains unsafe and data-inefficient. By fusing uncertainty-aware distributional outputs from each system, BCF arbitrates control between them, exploiting their respective strengths. We study BCF on two real-world robotics tasks involving navigation in a vast and long-horizon environment, and a complex reaching task that involves manipulability maximisation. For both these domains, there exist simple handcrafted controllers that can solve the task at hand in a risk-averse manner but do not necessarily exhibit the optimal solution given limitations in analytical modelling, controller miscalibration and task variation. As exploration is naturally guided by the prior in the early stages of training, BCF accelerates learning, while substantially improving beyond the performance of the control prior, as the policy gains more experience. More importantly, given the risk-aversity of the control prior, BCF ensures safe exploration and deployment, where the control prior naturally dominates the action distribution in states unknown to the policy. We additionally show BCF's applicability to the zeroshot sim-to-real setting and its ability to deal with out-ofdistribution states in the real-world. BCF is a promising approach for combining the complementary strengths of deep RL and traditional robotic control, surpassing what either can achieve independently. The code and supplementary video material are made publicly available at https://krishanrana.github.io/bcf.

show abstract

Section: Learned Controllers Classical Controllersmentioning

confidence: 99%

Section: Learned Controllers Classical Controllersmentioning

confidence: 99%

Section: Exploration Bias Using Behavioural Priorsmentioning

confidence: 99%

Section: A Tasksmentioning

confidence: 99%

Section: Evaluation Of Training Performancementioning

confidence: 99%

See 3 more Smart Citations

Bayesian Controller Fusion: Leveraging Control Priors in Deep Reinforcement Learning for Robotics

Rana¹,

Dasagi²,

Haviland³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

An Efficient Deep Reinforcement Learning Algorithm for Mapless Navigation with Gap-Guided Switching Strategy

Qin

Liu

et al. 2023

J Intell Robot Syst

View full text Add to dashboard Cite

Hybrid LMC: Hybrid Learning and Model-based Control for Wheeled Humanoid Robot via Ensemble Deep Reinforcement Learning

Baek

Purushottam

Ramos

2022

2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

Model-based controllers using a linearized model around the system's equilibrium point is a common approach in the control of a wheeled humanoid due to their less computational load and ease of stability analysis. However, controlling a wheeled humanoid robot while it lifts an unknown object presents significant challenges, primarily due to the lack of knowledge in object dynamics. This paper presents a framework designed for predicting the new equilibrium point explicitly to control a wheeled-legged robot with unknown dynamics. We estimated the total mass and center of mass of the system from its response to initially unknown dynamics, then calculated the new equilibrium point accordingly. To avoid using additional sensors (e.g., force torque sensor) and reduce the effort of obtaining expensive real data, a data-driven approach is utilized with a novel real-to-sim adaptation. A more accurate nonlinear dynamics model, offering a closer representation of real-world physics, is injected into a rigid-body simulation for real-to-sim adaptation. The nonlinear dynamics model parameters were optimized using Particle Swarm Optimization. The efficacy of this framework was validated on a physical wheeled inverted pendulum, a simplified model of a wheeledlegged robot. The experimental results indicate that employing a more precise analytical model with optimized parameters significantly reduces the gap between simulation and reality, thus improving the efficiency of a model-based controller in controlling a wheeled robot with unknown dynamics.

show abstract

Multiplicative Controller Fusion: Leveraging Algorithmic Priors for Sample-efficient Reinforcement Learning and Safe Sim-To-Real Transfer

Cited by 9 publications

References 16 publications

Bayesian Controller Fusion: Leveraging Control Priors in Deep Reinforcement Learning for Robotics

Bayesian Controller Fusion: Leveraging Control Priors in Deep Reinforcement Learning for Robotics

An Efficient Deep Reinforcement Learning Algorithm for Mapless Navigation with Gap-Guided Switching Strategy

Hybrid LMC: Hybrid Learning and Model-based Control for Wheeled Humanoid Robot via Ensemble Deep Reinforcement Learning

Contact Info

Product

Resources

About