Infinite Horizon Risk Sensitive Control of Discrete Time Markov Processes under Minorization Property

Masi, Giovanni B. Di; Stettner, Łukasz

doi:10.1137/040618631

Cited by 82 publications

(51 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Thus the eigenvector f * obtained in Corollary 6.1 is in P. It is straightforward to see that under condition (20), the conclusions of Lemma 5.3, Lemma 5.4 and Corollary 5.1 continue to be valid in this case, while the proofs require suitable modifications. In particular, the unique proper invariant set is…”

Section: Lemma 61mentioning

confidence: 85%

See 1 more Smart Citation

A Learning Algorithm for Risk-Sensitive Cost

2008

View full text Add to dashboard Cite

A linear function approximation based reinforcement learning algorithm is proposed for Markov decision processes with infinite horizon risk-sensitive cost. Its convergence is proved using the 'o.d.e. method' for stochastic approximation. The scheme is also extended to continuous state space processes. 1. Introduction. Recent decades have seen a major activity in approximate dynamic programming for Markov decision processes based on real or simulated data, using reinforcement learning algorithms. (See, e.g., Bertsekas and Tsitsiklis (1996) [10] and Sutton and Barto (1998) [30] for book length treatments and Si et al (2004) [28] for a flavour of more recent activity. While most of this work has focused on the additive cost criteria such as discounted or time-averaged cost, relatively little has been done for the multiplicative cost (or risk-sensitive cost as it is better known). There is, however, a lot of interest in this cost criterion, as it has important applications in finance, e.g., Bagchi and Sureshkumar (2002) (1983) [5] were developed. These were 'raw' in the sense that there was no explicit approximation of the value function in order to beat down the curse of dimensionality. In case of additive costs, there is a considerable body of work on such approximation architectures, one of the most popular being the linear function approximation. Here one seeks an approximation of the value function in terms of linear combination of a moderate number of basis functions specified a priori. The learning scheme then iteratively learns the weights (or coefficients) of this linear combination instead of learning the full value function, which is a much higher dimensional object. The first rigorous analysis of such a scheme is in Tsitsiklis and Van Roy (1997) [31], where its convergence was proved for the problem of policy evaluation. Since then there have been several variations of the basic theme, see, e.g., Bertsekas, Borkar and Nedic (2004) [8] and the references therein. The aim of this article is to propose a similar linear function approximation based learning scheme for policy evaluation in risk-sensitive control and justify it rigorously.

show abstract

Section: Lemma 61mentioning

confidence: 85%

“…This has been extensively studied, e.g., in Kontoyiannis and Meyn (2003) [25] and Kontoyiannis and Meyn (2005) [26]. See also the works of Di Masi and Stettner [20], [21] for risk-sensitive control on general state spaces, which also address this issue.…”

Section: Proof Writing (15) Asṙ(t) = H(r(t)) For a Suitably Defined mentioning

confidence: 99%

A Learning Algorithm for Risk-Sensitive Cost

2008

View full text Add to dashboard Cite

show abstract

“…This result follows from the proof of Proposition 1 of [2]. Namely by Remark 3 of [2] under (A3) the assumptions (B1) and (B3) of [2] are satisfied.…”

Section: Multiplicative Poisson Equationmentioning

confidence: 49%

“…The meaning of the splitting is explained in the following two results from [2] Lemma 2.1: Under the Markov control (a n ) ∈ U M the process (x n = (x 1 n , x 2 n )) is Markov with the transition operator P a n ,θ (x n , dy) defined by (i)-(iii) and if (a n ) ∈ U s , the process has a unique invariant measure Ψ (a n ),θ given by…”

Section: Splitting Of Markov Processesmentioning

confidence: 98%

“…In this paper a similar result is shown-that is, a construction is given for an off line adaptive control procedure for a more general class of controlled processes, namely the processes satisfying the minorization property (A1) and (A2). Risk sensitive control of such processes was considered in [2] and also in [3] and was a starting point for this research. There is a number of various important applications of the risk sensitive control, in particular in the mathematics of finance (see [1] and references therein).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Remarks on risk sensitive adaptive control of Markov processes

Duncan

Pasik-Duncan

Stettner

2006

Proceedings of the 45th IEEE Conference on Decision and Control

View full text Add to dashboard Cite

The study of adaptive control problems with multiplicative cost functionals, which appear in particular in risk sensitive control problems requires a different approach than that for risk neutral (average cost per unit time) problems. Assuming that the unknown parameter is selected from a known compact set, it is proved that there is a finite class of nearly optimal control functions for a parametrized family of risk sensitive control problems. Such a class is used to test the value of the cost functional over a finite time horizon, and the control function where the cost is largest is chosen to form an adaptive control. It is shown that this procedure is nearly optimal where the quality of this approximation depends on the choice of the nearly optimal controls and the length of the testing intervals.

show abstract

Solutions of the average cost optimality equation for finite Markov decision chains: risk-sensitive and risk-neutral criteria

Cavazos–Cadena

2008

Math Meth Oper Res

View full text Add to dashboard Cite

Infinite Horizon Risk Sensitive Control of Discrete Time Markov Processes under Minorization Property

Cited by 82 publications

References 16 publications

A Learning Algorithm for Risk-Sensitive Cost

A Learning Algorithm for Risk-Sensitive Cost

Remarks on risk sensitive adaptive control of Markov processes

Solutions of the average cost optimality equation for finite Markov decision chains: risk-sensitive and risk-neutral criteria

Contact Info

Product

Resources

About