A First Empirical Study of Emphatic Temporal Difference Learning

Ghiassian, Sina; Rafiee, Banafsheh; Sutton, Richard S.

doi:10.48550/arxiv.1705.04185

Cited by 2 publications

(5 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This experiment and the results on the Bertsekas's counterexample, together with several other results in [Ghiassian et al, 2017] and [Ghiassian et al, 2018], showed a consistent pattern of favoring Emphatic TD(λ) over conventional TD(λ) in the on-policy case. We did not strive to find a counterexample to challenge it.…”

Section: Discussionsupporting

confidence: 76%

“…A natural question to ask at this point is if on-policy ETD outperforms on-policy TD in all cases. We have seen evidences, the successful convergence of ETD on the Baird's counterexample ([Sutton and Barto, 2018]) as well as several other experiments did in ([Ghiassian et al, 2017]), that suggest ETD is a better algorithm than TD in the off-policy case. We do not know, however, whether a similar pattern carries through to the on-policy case.…”

Section: Possible Advantages Of Learning With Emphasismentioning

confidence: 57%

“…Surprisingly, ETD's on-policy version is also found to be significantly different from the conventional TD method. Yu's counterexample [Ghiassian et al, 2017] highlights this difference by providing a two-state on-policy problem in which TD diverged but ETD converged to the correct solution.…”

Section: Possible Advantages Of Learning With Emphasismentioning

confidence: 99%

“…The final counterexample from [Ghiassian et al, 2017] involves learning with variable λ's. Consider a two-state MRP as shown in the left panel of figure 3.…”

Section: Performance On Several Known On-policy Td Counterexamplesmentioning

confidence: 99%

See 3 more Smart Citations

Should All Temporal Difference Learning Use Emphasis?

Gu,

Ghiassian,

Sutton

2019

Preprint

Self Cite

View full text Add to dashboard Cite

Emphatic Temporal Difference (ETD) learning has recently been proposed as a convergent off-policy learning method. ETD was proposed mainly to address convergence issues of conventional Temporal Difference (TD) learning under offpolicy training but it is different from conventional TD learning even under on-policy training. A simple counterexample provided back in 2017 pointed to a potential class of problems where ETD converges but TD diverges. In this paper, we empirically show that ETD converges on a few other well-known on-policy experiments whereas TD either diverges or performs poorly. We also show that ETD outperforms TD on the mountain car prediction problem. Our results, together with a similar pattern observed under off-policy training in prior works, suggest that ETD might be a good substitute over conventional TD.

show abstract

Section: Discussionsupporting

confidence: 76%

Section: Possible Advantages Of Learning With Emphasismentioning

confidence: 57%

Section: Possible Advantages Of Learning With Emphasismentioning

confidence: 99%

“…The final counterexample from [Ghiassian et al, 2017] involves learning with variable λ's. Consider a two-state MRP as shown in the left panel of figure 3.…”

Section: Performance On Several Known On-policy Td Counterexamplesmentioning

confidence: 99%

See 2 more Smart Citations

Should All Temporal Difference Learning Use Emphasis?

Gu,

Ghiassian,

Sutton

2019

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…The idea of adding the penalty in the form of a regularization term has been considered in [11] where Regularized off-policy TD (RO-TD) algorithm has been proposed based on GTD algorithms and convex-concave saddle point formulations. Emphatic TD algorithms (ETD) [12]- [15] are another popular class of off-policy TD algorithms that achieve stability by emphasizing or de-emphasizing updates of the algorithm. These updates also have linear-time complexity.…”

Section: Introductionmentioning

confidence: 99%

A Convergent Off-Policy Temporal Difference Algorithm

Diddigi,

Kamanchi,

Bhatnagar

2019

Preprint

View full text Add to dashboard Cite

Learning the value function of a given policy (target policy) from the data samples obtained from a different policy (behavior policy) is an important problem in Reinforcement Learning (RL). This problem is studied under the setting of offpolicy prediction. Temporal Difference (TD) learning algorithms are a popular class of algorithms for solving the prediction problem. TD algorithms with linear function approximation are shown to be convergent when the samples are generated from the target policy (known as on-policy prediction). However, it has been well established in the literature that off-policy TD algorithms under linear function approximation diverge. In this work, we propose a convergent on-line off-policy TD algorithm under linear function approximation. The main idea is to penalize the updates of the algorithm in a way as to ensure convergence of the iterates. We provide a convergence analysis of our algorithm. Through numerical evaluations, we further demonstrate the effectiveness of our algorithm.

show abstract

A First Empirical Study of Emphatic Temporal Difference Learning

Cited by 2 publications

References 0 publications

Should All Temporal Difference Learning Use Emphasis?

Should All Temporal Difference Learning Use Emphasis?

A Convergent Off-Policy Temporal Difference Algorithm

Contact Info

Product

Resources

About