Deep dynamic policy programming for robot control with raw images

Tsurumine, Yoshihisa; Cui, Yunduan; Uchibe, Eiji; Matsubara, Takamitsu

doi:10.1109/iros.2017.8205960

Cited by 12 publications

(7 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A]. DPP has been extended to a deep learning setting [30], but it is less efficient than DQN 3 [31]. Taking the limit τ → 0, we retrieve Advantage Learning (AL) [5,7] (see Appx.…”

Section: What Happens Under the Hood?mentioning

confidence: 99%

Munchausen Reinforcement Learning

Vieillard,

Pietquin,

Geist

2020

Preprint

View full text Add to dashboard Cite

Bootstrapping is a core mechanism in Reinforcement Learning (RL). Most algorithms, based on temporal differences, replace the true value of a transiting state by their current estimate of this value. Yet, another estimate could be leveraged to bootstrap RL: the current policy. Our core contribution stands in a very simple idea: adding the scaled log-policy to the immediate reward. We show that slightly modifying Deep Q-Network (DQN) in that way provides an agent that is competitive with distributional methods on Atari games, without making use of distributional RL, n-step returns or prioritized replay. To demonstrate the versatility of this idea, we also use it together with an Implicit Quantile Network (IQN). The resulting agent outperforms Rainbow on Atari, installing a new State of the Art with very little modifications to the original algorithm. To add to this empirical study, we provide strong theoretical insights on what happens under the hood -implicit Kullback-Leibler regularization and increase of the action-gap.

show abstract

“…A]. DPP has been extended to a deep learning setting [30], but it is less efficient than DQN 3 [31]. Taking the limit τ → 0, we retrieve Advantage Learning (AL) [5,7] (see Appx.…”

Section: What Happens Under the Hood?mentioning

confidence: 99%

Munchausen Reinforcement Learning

Vieillard,

Pietquin,

Geist

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…From a practical point of view, neither SQL nor DPP have been originally implemented in RL on large scale problems. A deep version of a variation of DPP 4 have been proposed by Tsurumine et al [2017], but it is only applied on a small number of samples. The principal issue of a practical DPP is that it has to estimate ψ k , a quantity that is asymptotically unbounded.…”

Section: Related Work and Discussionmentioning

confidence: 99%

Momentum in Reinforcement Learning

Vieillard,

Scherrer,

Pietquin

et al. 2019

Preprint

View full text Add to dashboard Cite

We adapt the optimization's concept of momentum to reinforcement learning. Seeing the state-action value functions as an analog to the gradients in optimization, we interpret momentum as an average of consecutive q-functions. We derive Momentum Value Iteration (MoVI), a variation of Value Iteration that incorporates this momentum idea. Our analysis shows that this allows MoVI to average errors over successive iterations. We show that the proposed approach can be readily extended to deep learning. Specifically, we propose a simple improvement on DQN based on MoVI, and experiment it on Atari games.

show abstract

“…For the initial arm position, we use two different settings. The first (P 1) one uses a single initial arm position for learning as in [7]. This position is made well oriented for a palm-touching purpose.…”

Section: Experimental Protocolmentioning

confidence: 99%

“…1 F. de La Bourdonnaye, C.Teulière and T. Chateau are with the university of Clermont Auvergne, the Pascal Institute CNRS, UMR6602, Aubière, France 2 J.Triesch is with the Frankfurt Institute for Advanced Studies, Frankfurt am Main, Germany in manipulation robotics [3], [4], because they help to discriminate values of close states. For instance, [3], [4], [5], [6] and [7] use a distance measure between the current pose and a target pose to design an informative reward in manipulation tasks such as block stacking, reaching and door pushing or pulling. These rewards require knowledge of robot kinematics and target pose.…”

Section: Introductionmentioning

confidence: 99%

Learning to Touch Objects Through Stage-Wise Deep Reinforcement Learning

Bourdonnaye

Teulière

Triesch

et al. 2018

2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

Learning complex behaviors through reinforcement learning is particularly challenging when reward is only available upon successful completion of the full behavior. In manipulation robotics, so-called shaping rewards are often used to overcome this problem. However, these usually require human engineering or (partial) world models describing, e.g., the kinematics of the robot or high-level modules for perception. Here we propose an alternative method to learn an object palm-touching task through a weakly-supervised and stagewise learning of simpler tasks. First, the robot learns to fixate the object with its cameras. Second, the robot learns eye-hand coordination by learning to fixate its end effector. Third, using the previously acquired skills an informative shaping reward can be computed which facilitates efficient learning of the object palm-touching task. We demonstrate in simulation that learning the full task with this shaping reward is comparable to learning with an informative supervised reward.

show abstract

Deep dynamic policy programming for robot control with raw images

Cited by 12 publications

References 14 publications

Munchausen Reinforcement Learning

Munchausen Reinforcement Learning

Momentum in Reinforcement Learning

Learning to Touch Objects Through Stage-Wise Deep Reinforcement Learning

Contact Info

Product

Resources

About