Direct Policy Gradients: Direct Optimization of Policies in Discrete Action Spaces

Lorberbom, Guy; Maddison, Chris J.; Heess, Nicolas; Hazan, Tamir; Tarlow, Daniel

doi:10.48550/arxiv.1906.06062

Search citation statements

Order By: Relevance

Paper Sections

Select...

The Unordered Set Policy Gradient Estimator1

Citation Types

Supporting

Mentioning

Contrasting

Year Published

2020

Publication Types

Select...

Other1

Relationship

Self Cite0

Independent1

Authors

Journals

Cited by 1 publication

(1 citation statement)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This includes sequence models, from which we can sample using Stochastic Beam Search (Kool et al, 2019c), as well as multivariate categorical distributions which can also be treated as sequence models (see Section 4.2). In the presence of continuous variables or a stochastic function f , we may separate this stochasticity from the stochasticity over the discrete distribution, as in Lorberbom et al (2019). The computation of the leave-one-out ratios adds some overhead, although they can be computed efficiently, even for large k (see Appendix B).…”

Section: The Unordered Set Policy Gradient Estimatormentioning

confidence: 99%

Estimating Gradients for Discrete Random Variables by Sampling without Replacement

Kool¹,

Hoof²,

Welling³

2020

Preprint

View full text Add to dashboard Cite

We derive an unbiased estimator for expectations over discrete random variables based on sampling without replacement, which reduces variance as it avoids duplicate samples. We show that our estimator can be derived as the Rao-Blackwellization of three different estimators. Combining our estimator with RE-INFORCE, we obtain a policy gradient estimator and we reduce its variance using a built-in control variate which is obtained without additional model evaluations. The resulting estimator is closely related to other gradient estimators. Experiments with a toy problem, a categorical Variational Auto-Encoder and a structured prediction problem show that our estimator is the only estimator that is consistently among the best estimators in both high and low entropy settings.

show abstract

Section: The Unordered Set Policy Gradient Estimatormentioning

confidence: 99%