Information Relaxation and Dual Formulation of Controlled Markov Diffusions

Ye, Fan; Zhou, Enlu

doi:10.1109/tac.2015.2418672

Cited by 14 publications

(10 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These papers develop methods for bounding the performance of heuristic policies relative to the optimal for non-worst-case stochastic control problems. ( Haugh andYe andZhou 2015 for related ideas. ) In this regard, one contribution of this paper is that it develops systematic performance evaluation methods for worst-case robust control problems, which we apply to performance evaluation of the robust Gittins index.…”

Section: Relevant Literaturementioning

confidence: 99%

“…The lower bound V L x is obtained in §5.3 by adapting the idea of information relaxation (Brown et al 2010, Haugh and Lim 2012, Rogers 2007, Ye and Zhou 2015, which relaxes the requirement that nature's actions depend only on the information available at the time of the decision. This can be viewed as a "strengthening of nature," because it gives nature the ability to make use of future state information.…”

Section: Performance Evaluationmentioning

confidence: 99%

See 1 more Smart Citation

Robust Multiarmed Bandit Problems

Kim

Lim

2016

Management Science

View full text Add to dashboard Cite

T he multiarmed bandit problem is a popular framework for studying the exploration versus exploitation trade-off. Recent applications include dynamic assortment design, Internet advertising, dynamic pricing, and the control of queues. The standard mathematical formulation for a bandit problem makes the strong assumption that the decision maker has a full characterization of the joint distribution of the rewards, and that "arms" under this distribution are independent. These assumptions are not satisfied in many applications, and the outof-sample performance of policies that optimize a misspecified model can be poor. Motivated by these concerns, we formulate a robust bandit problem in which a decision maker accounts for distrust in the nominal model by solving a worst-case problem against an adversary ("nature") who has the ability to alter the underlying reward distribution and does so to minimize the decision maker's expected total profit. Structural properties of the optimal worst-case policy are characterized by using the robust Bellman (dynamic programming) equation, and arms are shown to be no longer independent under nature's worst-case response. One implication of this is that index policies are not optimal for the robust problem, and we propose, as an alternative, a robust version of the Gittins index. Performance bounds for the robust Gittins index are derived by using structural properties of the value function together with ideas from stochastic dynamic programming duality. We also investigate the performance of the robust Gittins index policy when applied to a Bayesian webpage design problem. In the presence of model misspecification, numerical experiments show that the robust Gittins index policy not only outperforms the classical Gittins index policy, but also substantially reduces the variability in the out-of-sample performance.

show abstract

Section: Relevant Literaturementioning

confidence: 99%

Section: Performance Evaluationmentioning

confidence: 99%

Robust Multiarmed Bandit Problems

Kim

Lim

2016

Management Science

View full text Add to dashboard Cite

show abstract

“…Here we present a simpler version of their approach so that its connection with our proposed approach is more clear. Ye and Zhou (2015) studies the form of optimal dual penalty under the setting of controlled Markov diffusion (CMD) and show that it is a stochastic integral. It then inspires the authors to propose an approximation scheme for the optimal dual penalty of a discrete-time DP with state dynamics (3.14) as follows:…”

Section: American Option Pricingmentioning

confidence: 99%

“…Our framework is more universal and powerful, because it reveals the structure of the optimal dual penalty regardless of the underlying probability measure (i.e., not restricted to the Brownian measure in Belomestny et al (2009) or the Poisson random measure in Zhu et al (2015)). We will also show that the approximation scheme in Ye and Zhou (2015) could be viewed as a special case of the proposed framework as well. To summarize, the contributions are as follows:…”

Section: Introductionmentioning

confidence: 98%

Solving the Dual Problems of Dynamic Programs via Regression

Zhu

Zhou

2018

IEEE Trans. Automat. Contr.

Self Cite

View full text Add to dashboard Cite

In recent years, information relaxation and duality in dynamic programs have been studied extensively, and the resulted primal-dual approach has become a powerful procedure in solving dynamic programs by providing lower-upper bounds on the optimal value function. Theoretically, with the so called value-based optimal dual penalty, the optimal value function could be recovered exactly via strong duality. However, in practice, obtaining tight dual bounds usually requires good approximations of the optimal dual penalty, which could be time-consuming due to the conditional expectations that need to be estimated via nested simulation. In this paper, we will develop a framework of regression approach to approximating the optimal dual penalty in a non-nested manner, by exploring the structure of the function space consisting of all feasible dual penalties. The resulted approximations maintain to be feasible dual penalties, and thus yield valid dual bounds on the optimal value function. We show that the proposed framework is computationally efficient, and the resulted dual penalties lead to numerically tractable dual problems. Finally, we apply the framework to a high-dimensional dynamic trading problem to demonstrate its effectiveness in solving the dual problems of complex dynamic programs.

show abstract

“…Another constraint is the "information constraint" or non-anticipativity of the control policy, that is, the decision should depend on the information up to the time that the decision is made. These relaxations may lead to a simpler dynamic optimization problem: the first constraint that exists universally in mathematical programs can be tackled by the commonly known Lagrangian relaxation (see, e.g., [5]), which results in an unconstrained stochastic dynamic program that may be easier to solve; the second constraint can be approached by a recently developed technique -"information relaxation" (see, e.g, [6], [7], [8], [9], [10]), which relaxes the non-anticipativity constraint on the controls but impose a penalty for such a violation. Since this approach allows the decision to be made based on the future outcome, it involves scenario-based dynamic programs, which are deterministic optimization problems and may be less complicated than the original stochastic dynamic program.…”

mentioning

confidence: 99%

Weakly Coupled Dynamic Program: Information and Lagrangian Relaxations

Zhu

Zhou

2018

IEEE Trans. Automat. Contr.

Self Cite

View full text Add to dashboard Cite

Weakly coupled dynamic program" describes a broad class of stochastic optimization problems in which multiple controlled stochastic processes evolve independently but subject to a set of linking constraints imposed on the controls. One feature of the weakly coupled dynamic program is that it decouples into lower-dimensional dynamic programs by dualizing the linking constraint via the Lagrangian relaxation, which also yields a bound on the optimal value of the original dynamic program. Together with the Lagrangian bound, we utilize the information relaxation approach that relaxes the non-anticipative constraint on the controls to obtain a tighter dual bound. We also investigate other combinations of the relaxations and place the resulting bounds in order. To tackle large-scale problems, we further propose a computationally tractable method based on information relaxation, and provide insightful interpretation and performance guarantee. We implement our method and demonstrate its use through two numerical examples.

show abstract

Information Relaxation and Dual Formulation of Controlled Markov Diffusions

Cited by 14 publications

References 44 publications

Robust Multiarmed Bandit Problems

Robust Multiarmed Bandit Problems

Solving the Dual Problems of Dynamic Programs via Regression

Weakly Coupled Dynamic Program: Information and Lagrangian Relaxations

Contact Info

Product

Resources

About