Generalized Emphatic Temporal Difference Learning: Bias-Variance Analysis

Hallak, Assaf; Tamar, Aviv; Munos, Rémi; Mannor, Shie

doi:10.1609/aaai.v30i1.10227

Cited by 8 publications

(9 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As previously described for goal oriented RL, contextual RL aims to solve problems known as Contextual MDP (CMDP) [16]. The first proposed version of CMDP in [17] defines context as augmentation of MDP using side information as form of context, in a similar fashion to contextual bandits. Contextual MDP (CMDP) can be defined as follows:…”

Section: Contextual Rlmentioning

confidence: 99%

Learning intraoperative organ manipulation with context-based reinforcement learning

et al. 2022

View full text Add to dashboard Cite

Purpose Automation of sub-tasks during robotic surgery is challenging due to the high variability of the surgical scenes intra- and inter-patients. For example, the pick and place task can be executed different times during the same operation and for distinct purposes. Hence, designing automation solutions that can generalise a skill over different contexts becomes hard. All the experiments are conducted using the Pneumatic Attachable Flexible (PAF) rail, a novel surgical tool designed for robotic-assisted intraoperative organ manipulation. Methods We build upon previous open-source surgical Reinforcement Learning (RL) training environment to develop a new RL framework for manipulation skills, rlman. In rlman, contextual RL agents are trained to solve different aspects of the pick and place task using the PAF rail system. rlman is implemented to support both low- and high-dimensional state information to solve surgical sub-tasks in a simulation environment. Results We use rlman to train state of the art RL agents to solve four different surgical sub-tasks involving manipulation skills using the PAF rail. We compare the results with state-of-the-art benchmarks found in the literature. We evaluate the ability of the agent to be able to generalise over different aspects of the targeted surgical environment. Conclusion We have shown that the rlman framework can support the training of different RL algorithms for solving surgical sub-task, analysing the importance of context information for generalisation capabilities. We are aiming to deploy the trained policy on the real da Vinci using the dVRK and show that the generalisation of the trained policy can be transferred to the real world.

show abstract

Section: Contextual Rlmentioning

confidence: 99%

Learning intraoperative organ manipulation with context-based reinforcement learning

et al. 2022

View full text Add to dashboard Cite

show abstract

“…Motivation: The inspiration for the algorithm comes from the work of Hallak et al [26]. Hallak et al [26] demonstrate that by controlling β in the ETD(λ, β) algorithm, the variance of the algorithm can be reduced while still maintaining a reasonable bias, thus allowing for improved performance. The vanilla GEM algorithm for emphasis estimation and its variant GEM-ETD for OPE truly trade off bias and variance.…”

Section: Proposed Emphatic Algorithmsmentioning

confidence: 99%

“…The ETD approach, which was originally proposed in the pioneering work of Sutton et al [1], involves a scalar followon trace to surmount the distribution mismatch issue with off-policy learning. On the other hand, Hallak et al [26] demonstrated that the variance of the followon trace can be unbounded over a long or infinite time horizon. They further proposed the ETD(0, β) framework to bound the followon trace with a tunable hyperparameter but at the cost of a possibly large bias error, where β is a variable decay rate used for bias-variance trade-off.…”

Section: Generalized Gradient Emphasis Learning With Function Approxi...mentioning

confidence: 99%

“…Consider the two-state Markov chain shown in Fig. 1(a), which is an off-policy counterexample presented by Kolter [52] to show the bias bound of the off-policy TD, and used by Hallak et al [26] to demonstrate the bias bound of the ETD. In this MDP model, the transition probability matrix We benchmark the vanilla TD, the vanilla ETD(β), and GEM-ETD(β) in this off-policy counterexample.…”

Section: Fixed Points Analysismentioning

confidence: 99%

“…In this paper, we focus on emphatic methods, and how to better learn the emphasis and derive more efficient OPE and off-policy control algorithms is the focus of our work. Thus, in this work, we draw inspiration from Hallak et al [26] and develop a flexible and general algorithm for the biasvariance trade-off of the emphasis learning to alleviate the problem-dependent dilemma. The contributions of our paper are as follows:…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Generalized Gradient Emphasis Learning for Off-Policy Evaluation and Control with Function Approximation

Cao¹,

Liu²,

Wu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Emphatic temporal-difference (TD) learning (Sutton et al. 2016) is a pioneering off-policy reinforcement learning method involving the use of the followon trace. The recently proposed Gradient Emphasis Learning (GEM, Zhang et al. 2020) algorithm is used to fix the problems of unbounded variance and large emphasis approximation error introduced by the followon trace from the perspective of stochastic approximation. In this paper, we rethink GEM and introduce a novel generalized GEM(β) algorithm to learn the true emphasis. The key to the construction of the generalized GEM(β) algorithm is introducing a tunable hyper-parameter β that is not necessarily the same as the discount factor γ to the GEM operator. We then apply the emphasis estimated by the proposed GEM(β) algorithm to the value estimation gradient and the policy gradient, respectively, yielding the corresponding emphatic TD variant for off-policy evaluation and actor-critic algorithm for off-policy control. Finally, we demonstrate empirically the advantage of the proposed algorithms across a range of problems, for both off-policy evaluation and off-policy control, and for both linear and nonlinear function approximation.

show abstract

Q($$\lambda $$) with Off-Policy Corrections

Harutyunyan¹,

Bellemare

Stepleton

et al. 2016

Lecture Notes in Computer Science

View full text Add to dashboard Cite

We propose and analyze an alternate approach to off-policy multi-step temporal difference learning, in which off-policy returns are corrected with the current Q-function in terms of rewards, rather than with the target policy in terms of transition probabilities. We prove that such approximate corrections are sufficient for off-policy convergence both in policy evaluation and control, provided certain conditions. These conditions relate the distance between the target and behavior policies, the eligibility trace parameter and the discount factor, and formalize an underlying tradeoff in off-policy TD(λ). We illustrate this theoretical relationship empirically on a continuous-state control task.

show abstract

Generalized Emphatic Temporal Difference Learning: Bias-Variance Analysis

Cited by 8 publications

References 9 publications

Learning intraoperative organ manipulation with context-based reinforcement learning

Learning intraoperative organ manipulation with context-based reinforcement learning

Generalized Gradient Emphasis Learning for Off-Policy Evaluation and Control with Function Approximation

Q($$\lambda $$) with Off-Policy Corrections

Contact Info

Product

Resources

About