Optimal learning with non-Gaussian rewards

Ding, Zi; Ryzhov, Ilya O.

doi:10.1109/wsc.2013.6721457

Cited by 3 publications

(2 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…First, the rate of convergence is optimized when the exact OCBA ratios are used, but this is not guaranteed if those ratios are only achieved asymptotically. Second, it is likely that the normality assumptions play a key role in the equivalence, since EI-type methods may not even be consistent in non-normal settings (Ding and Ryzhov 2015). Nonetheless, normality assumptions remain widely used in the literature and in practice, and it may be possible to obtain similar results for other learning problems where such assumptions are made.…”

Section: Resultsmentioning

confidence: 99%

Expected improvement is equivalent to OCBA

Ryzhov¹

2015

2015 Winter Simulation Conference (WSC)

Self Cite

View full text Add to dashboard Cite

This paper summarizes new theoretical results on the asymptotic sampling rates of expected improvement (EI) methods in fully sequential ranking and selection (R&S). These methods have been widely observed to perform well in practice, and often have asymptotic consistency properties, but rate results are generally difficult to obtain when observations are subject to stochastic noise. We find that, in one general R&S problem, variants of EI produce simulation allocations that are virtually identical to the rate-optimal allocations calculated by the optimal computing budget allocation (OCBA) methodology. This result provides new insight into the good empirical performance of EI under normality assumptions.

show abstract

Section: Resultsmentioning

confidence: 99%

Expected improvement is equivalent to OCBA

Ryzhov¹

2015

2015 Winter Simulation Conference (WSC)

Self Cite

View full text Add to dashboard Cite

show abstract

“…They observe that ν KG a can be zero, but do not appear to recognize that this can yield dominated actions under the policy. Later work (Ding and Ryzhov [4]) showed that this can lead to the offline KG policy never choosing the greedy arm, an extreme case of dominated errors. However, with the online KG policy the greedy arm will eventually be selected as ν KG a for the other arm tends to zero.…”

Section: Exponential Rewardsmentioning

confidence: 99%

On the Identification and Mitigation of Weaknesses in the Knowledge Gradient Policy for Multi-Armed Bandits

Edwards¹,

Fearnhead²,

Glazebrook³

2016

Prob. Eng. Inf. Sci.

View full text Add to dashboard Cite

The Knowledge Gradient (KG) policy was originally proposed for offline ranking and selection problems but has recently been adapted for use in online decisionmaking in general and multi-armed bandit problems (MABs) in particular. We study its use in a class of exponential family MABs and identify weaknesses, including a propensity to take actions which are dominated with respect to both exploitation and exploration. We propose variants of KG which avoid such errors. These new policies include an index heuristic which deploys a KG approach to develop an approximation to the Gittins index. A numerical study shows this policy to perform well over a range of MABs including those for which index policies are not optimal. While KG does not take dominated actions when bandits are Gaussian, it fails to be index consistent and appears not to enjoy a performance advantage over competitor policies when arms are correlated to compensate for its greater computational demands.arXiv:1607.05970v2 [stat.ML]

show abstract

Optimal learning with non-Gaussian rewards

Ding¹,

Ryzhov²

2013

2013 Winter Simulations Conference (WSC)

View full text Add to dashboard Cite

In this disseration, the author studies sequential Bayesian learning problems modeled under non-Gaussian distributions. We focus on a class of problems called the multi-armed bandit problem, and studies its optimal learning strategy, the Gittins index policy. The Gittins index is computationally intractable and approximation methods have been developed for Gaussian reward problems. We construct a novel theoretical and computational framework for the Gittins index under nonGaussian rewards. By interpolating the rewards using continuous-time conditional Lévy processes, we recast the optimal stopping problems that characterize Gittins indices into free-boundary partial integro-differential equations (PIDEs). We also provide additional structural properties and numerical illustrations on how our approach can be used to approximate the Gittins index.Optimal Learning with Non-Gaussian Rewards

show abstract

Optimal learning with non-Gaussian rewards

Cited by 3 publications

References 40 publications

Expected improvement is equivalent to OCBA

Expected improvement is equivalent to OCBA

On the Identification and Mitigation of Weaknesses in the Knowledge Gradient Policy for Multi-Armed Bandits

Optimal learning with non-Gaussian rewards

Contact Info

Product

Resources

About