Sample-based Distributional Policy Gradient

Singh, Rahul; Lee, Keuntaek; Chen, Yongxin

doi:10.48550/arxiv.2001.02652

Cited by 3 publications

(10 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Since the successful implementation of RL problems from a distributional perspective on Atari 2600 games (Bellemare et al, 2017a), there is a number of follow-ups trying to boost existing deep RL algorithms by directly characterizing the distribution of the random return instead of the expectation (Dabney et al, 2018a,b;Barth-Maron et al, 2018;Singh et al, 2020).…”

Section: Related Workmentioning

confidence: 99%

“…On the policy-gradient-based side, D4PG (Barth-Maron et al, 2018) incorporates the distributional perspective into DDPG (Lillicrap et al, 2015), with the return distribution modeled similarly as in C51 (Bellemare et al, 2017a). On top of that, SDPG (Singh et al, 2020) is proposed to model the quantile function with a generator to overcome the limitation of using variable probabilities at fixed locations, and the same as D4PG, it models the policy as a deterministic transformation of the state representation.…”

Section: Related Workmentioning

confidence: 99%

“…(c): Similar to Singh et al (2020), we check the matching situation of minimizing the quantile regression Huber loss. In detail, we generate 10000 random noises k ∼ p( ) to obtain {G ω1 (s, a, k )} 10000 k=1 and {r(s, a) + γG ω1 (s , a , k )} 10000 k=1 , and then compare their histograms to check if the empirical distributions are similar to each other.…”

Section: Evaluation Study To Answer Questions (A)-(c)mentioning

confidence: 99%

“…1 SAC-J refers to SAC with J actions to estimate its objective function. 2 Note that the SDPG paper (Singh et al, 2020) is using a different codebase; the implementation-level differences make their reported results not directly comparable; we use this variant to illustrate how each component works.…”

Section: Ablation Study To Answer Questions (D)-(g)mentioning

confidence: 99%

“…Later on, the distributional view is also incorporated into the framework of deep deterministic policy gradient (DDPG) (Lillicrap et al, 2015) for continuous control tasks, yielding efficient algorithms such as distributed distributional DDPG (D4PG) (Barth-Maron et al, 2018) and sample-based distributional policy gradient (SDPG) (Singh et al, 2020). Due to the deterministic nature of the policy, these algorithms always manually add random noises to actions during the training process to avoid getting stuck in poor local optimums.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Implicit Distributional Reinforcement Learning

Yue¹,

Wang²,

Zhou³

2020

Preprint

View full text Add to dashboard Cite

To improve the sample efficiency of policy-gradient based reinforcement learning algorithms, we propose implicit distributional actor critic (IDAC) that consists of a distributional critic, built on two deep generator networks (DGNs), and a semi-implicit actor (SIA), powered by a flexible policy distribution. We adopt a distributional perspective on the discounted cumulative return and model it with a state-action-dependent implicit distribution, which is approximated by the DGNs that take state-action pairs and random noises as their input. Moreover, we use the SIA to provide a semi-implicit policy distribution, which mixes the policy parameters with a reparameterizable distribution that is not constrained by an analytic density function. In this way, the policy's marginal distribution is implicit, providing the potential to model complex properties such as covariance structure and skewness, but its parameter and entropy can still be estimated. We incorporate these features with an off-policy algorithm framework to solve problems with continuous action space, and compare IDAC with the state-of-art algorithms on representative OpenAI Gym environments. We observe that IDAC outperforms these baselines for most tasks.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Evaluation Study To Answer Questions (A)-(c)mentioning

confidence: 99%

Section: Ablation Study To Answer Questions (D)-(g)mentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Implicit Distributional Reinforcement Learning

Yue¹,

Wang²,

Zhou³

2020

Preprint

View full text Add to dashboard Cite

show abstract

Improving Robustness via Risk Averse Distributional Reinforcement Learning

Singh¹,

Zhang²,

Chen³

2020

Preprint

Self Cite

View full text Add to dashboard Cite

One major obstacle that precludes the success of reinforcement learning in real-world applications is the lack of robustness, either to model uncertainties or external disturbances, of the trained policies. Robustness is critical when the policies are trained in simulations instead of real world environment. In this work, we propose a risk-aware algorithm to learn robust policies in order to bridge the gap between simulation training and real-world implementation. Our algorithm is based on recently discovered distributional RL framework. We incorporate CVaR risk measure in sample based distributional policy gradients (SDPG) for learning risk-averse policies to achieve robustness against a range of system disturbances. We validate the robustness of risk-aware SDPG on multiple environments.

show abstract

Sample-based Uncertainty Quantification with a Single Deterministic Neural Network

Kanazawa¹,

Gupta²

2022

Preprint

View full text Add to dashboard Cite

Development of an accurate, flexible, and numerically efficient uncertainty quantification (UQ) method is one of fundamental challenges in machine learning. Previously, a UQ method called DISCO Nets has been proposed (Bouchacourt et al., 2016) that trains a neural network by minimizing the so-called energy score on training data. This method has shown superior performance on a hand pose estimation task in computer vision, but it remained unclear whether this method works as nicely for regression on tabular data, and how it competes with more recent advanced UQ methods such as NGBoost. In this paper, we propose an improved neural architecture of DISCO Nets that admits a more stable and smooth training. We benchmark this approach on miscellaneous real-world tabular datasets and confirm that it is competitive with or even superior to standard UQ baselines. We also provide a new elementary proof for the validity of using the energy score to learn predictive distributions. Further, we point out that DISCO Nets in its original form ignore epistemic uncertainty and only capture aleatoric uncertainty. We propose a simple fix to this problem.

show abstract

Sample-based Distributional Policy Gradient

Cited by 3 publications

References 11 publications

Implicit Distributional Reinforcement Learning

Implicit Distributional Reinforcement Learning

Improving Robustness via Risk Averse Distributional Reinforcement Learning

Sample-based Uncertainty Quantification with a Single Deterministic Neural Network

Contact Info

Product

Resources

About