Adaptive Pairwise Weights for Temporal Credit Assignment

Zheng, Zhihua; Vuorio, Risto; Lewis, Richard A.; Singh, Satinder

doi:10.48550/arxiv.2102.04999

Cited by 1 publication

(1 citation statement)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The simplest example is studying how the choice of discount factor γ affects the policy learning (Petrik & Scherrer, 2008;Jiang et al, 2015;Fedus et al, 2019). Several previous work consider to extend the λ-return mechanism (Sutton, 1988) to a more generalized credit assignment framework, such as adaptive λ (Xu et al, 2018) and pairwise weights (Zheng et al, 2021). RUDDER (Arjona-Medina et al, 2019) proposes a return-equivalent formulation for the credit assignment problem and establish theoretical analyses (Holzleitner et al, 2021).…”

Section: Related Workmentioning

confidence: 99%

Learning Long-Term Reward Redistribution via Randomized Return Decomposition

Ren¹,

Guo²,

Zhou³

et al. 2021

Preprint

View full text Add to dashboard Cite

Many practical applications of reinforcement learning require agents to learn from sparse and delayed rewards. It challenges the ability of agents to attribute their actions to future outcomes. In this paper, we consider the problem formulation of episodic reinforcement learning with trajectory feedback. It refers to an extreme delay of reward signals, in which the agent can only obtain one reward signal at the end of each trajectory. A popular paradigm for this problem setting is learning with a designed auxiliary dense reward function, namely proxy reward, instead of sparse environmental signals. Based on this framework, this paper proposes a novel reward redistribution algorithm, randomized return decomposition (RRD), to learn a proxy reward function for episodic reinforcement learning. We establish a surrogate problem by Monte-Carlo sampling that scales up least-squares-based reward redistribution to long-horizon problems. We analyze our surrogate loss function by connection with existing methods in the literature, which illustrates the algorithmic properties of our approach. In experiments, we extensively evaluate our proposed method on a variety of benchmark tasks with episodic rewards and demonstrate substantial improvement over baseline algorithms.

show abstract

Section: Related Workmentioning

confidence: 99%