Markov control processes with pathwise constraints

Mendoza-Pérez, Armando F.; Hernández–Lerma, Onésimo

doi:10.1007/s00186-010-0311-8

Cited by 12 publications

(19 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(i) (see [9,Theorem 3.4]) There exists an optimal policy ' Ã 2 È for the pathwise CP (5), (6). In particular, g c (' Ã ) and g('…”

Section: Remark 33 By Assumptions 32(a) and 31(b3) We Have Thatmentioning

confidence: 98%

“…The works [8][9][10][13][14][15] have shown the existence of optimal policies for several classes of MCPs with pathwise constraints, that is, policies that maximize a certain long-run pathwise average reward subjet to a constraint on a long-run pathwise average cost. In [10], we give conditions for the existence of deterministic (as opposed to randomized) optimal policies for the pathwise CP.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Variance-minimization of Markov control processes with pathwise constraints

Mendoza-Pérez

Hernández–Lerma

2012

Optimization

Self Cite

View full text Add to dashboard Cite

show abstract

“…(i) (see [9,Theorem 3.4]) There exists an optimal policy ' Ã 2 È for the pathwise CP (5), (6). In particular, g c (' Ã ) and g('…”

Section: Remark 33 By Assumptions 32(a) and 31(b3) We Have Thatmentioning

confidence: 98%

Section: Introductionmentioning

confidence: 99%

Variance-minimization of Markov control processes with pathwise constraints

Mendoza-Pérez

Hernández–Lerma

2012

Optimization

Self Cite

View full text Add to dashboard Cite

show abstract

“…Sufficient conditions for this Assumption are given, for instance, in Assumption 4.1 and Lemma 4.8 of [28].…”

Section: Assumption 28mentioning

confidence: 99%

Policy Iteration Algorithms for Zero-Sum Stochastic Differential Games with Long-Run Average Payoff Criteria

López-Barrientos

2014

J. Oper. Res. Soc. China

View full text Add to dashboard Cite

This paper studies the policy iteration algorithm (PIA) for zero-sum stochastic differential games with the basic long-run average criterion, as well as with its more selective version, the so-called bias criterion. The system is assumed to be a nondegenerate diffusion. We use Lyapunov-like stability conditions that ensure the existence and boundedness of the solution to certain Poisson equation. We also ensure the convergence of a sequence of such solutions, of the corresponding sequence of policies, and, ultimately, of the PIA.

show abstract

“…The special case of constraints on expected utility in discounted MDPs is considered in [29]. MDPs with expected constraints and pathwise constraints, also called hard constraints, are considered in [32] using convex analytic methods. An inventory system is detailed to motivate the theoretical results.…”

Section: Introductionmentioning

confidence: 99%

Stochastic Dominance-Constrained Markov Decision Processes

Haskell¹,

Jain²

2013

SIAM J. Control Optim.

View full text Add to dashboard Cite

We are interested in risk constraints for infinite horizon discrete time Markov decision processes (MDPs). Starting with average reward MDPs, we show that increasing concave stochastic dominance constraints on the empirical distribution of reward lead to linear constraints on occupation measures. The optimal policy for the resulting class of dominance-constrained MDPs is obtained by solving a linear program. We compute the dual of this linear program to obtain average dynamic programming optimality equations that reflect the dominance constraint. In particular, a new pricing term appears in the optimality equations corresponding to the dominance constraint. We show that many types of stochastic orders can be used in place of the increasing concave stochastic order. We also carry out a parallel development for discounted reward MDPs with stochastic dominance constraints. The paper concludes with a portfolio optimization example.Risk management for MDPs has been considered from many perspectives in the literature. [20] includes penalties for the variance of rewards in MDPs. The optimal policy is obtained by solving a nonlinear programming problem in occupation measures. In [37], the mean-variance trade-off in MDPs is further explored in a Pareto-optimality sense. The conditional value-at-risk of the total cost in a finite horizon MDPs is constrained in [4]. It is argued that convex analytic methods do not apply to this problem type and an offline iterative algorithm is employed to solve for the optimal policy. [35] develops Markov risk measures for finite horizon and infinite horizon discounted MDPs. Dynamic programming equations are derived that reflect the risk aversion, and policy iteration is shown to solve the infinite horizon problem.Our notion of risk constrained MDPs differs from this literature survey. We are interested in the empirical distribution of reward, rather than in its expectation, variance, or other summary statistics. Our approach is based on stochastic orders, which are partial orders on the space of random variables, see [33,36] for extensive monographs on stochastic orders. [9, 10] use the increasing concave stochastic order to define stochastic dominance constraints in single stage stochastic optimization. The increasing concave stochastic order is notable for its connection to risk-averse decision makers, i.e. it captures the preferences of all risk-averse decision makers. A benchmark random variable is introduced, and a concave random variable-valued mapping is constrained to dominate the benchmark in the increasing concave stochastic order. It is shown that increasing concave functions are the Lagrange multipliers of the dominance constraints. The dual problem is a search over a certain class of increasing concave functions, interpreted as utility functions, and strong duality is established. Stochastic dominance constraints are applied to finite horizon stochastic programming problems with linear system dynamics in [12]. Specifically, a stochastic dominance constraint is placed on a vec...

show abstract

Markov control processes with pathwise constraints

Cited by 12 publications

References 22 publications

Variance-minimization of Markov control processes with pathwise constraints

Variance-minimization of Markov control processes with pathwise constraints

Policy Iteration Algorithms for Zero-Sum Stochastic Differential Games with Long-Run Average Payoff Criteria

Stochastic Dominance-Constrained Markov Decision Processes

Contact Info

Product

Resources

About