Risk-Sensitive Markov Decision Processes

Howard, Ronald A.; Matheson, James

doi:10.1287/mnsc.18.7.356

Cited by 432 publications

(305 citation statements)

References 1 publication

Supporting

Mentioning

259

Contrasting

Unclassified

Order By: Relevance

“…Barz and Waldmann [6] introduce an exponential utility function and employ the results of Howard and Matheson [25] in order to derive an opti-mal policy for this approach. An exponential utility function has the form u γ (x) = − exp(−γx) with positive parameter γ determining the level of risk aversion.…”

Section: Exponential Utility Functionmentioning

confidence: 99%

Risk management policies for dynamic capacity control

Koenig

Meissner

2015

Computers & Operations Research

View full text Add to dashboard Cite

Consider a dynamic decision making model under risk with a fixed planning horizon, namely the dynamic capacity control model. The model describes a firm, operating in a monopolistic setting and selling a range of products consuming a single resource. Demand for each product is time-dependent and modeled by a random variable. The firm controls the revenue stream by allowing or denying customer requests for product classes. We investigate risk-sensitive policies in this setting, for which risk concerns are important for many non-repetitive events and short-time considerations.Numerically analyzing several risk-averse capacity control policies in terms of standard deviation and conditional-value-at-risk, our results show that only a slight modification of the risk-neutral solution is needed to apply a risk-averse policy. In particular, risk-averse policies which decision rules are functions depending only on the marginal values of the riskneutral policy perform well. From a practical perspective, the advantage is that a decision maker does not need to compute any risk-averse dynamic program. Risk sensitivity can be easily achieved by implementing risk-averse functional decision rules based on a risk-neutral solution.

show abstract

Section: Exponential Utility Functionmentioning

confidence: 99%

Risk management policies for dynamic capacity control

Koenig

Meissner

2015

Computers & Operations Research

View full text Add to dashboard Cite

show abstract

“…In relation to other risk-sensitive approaches, such as the ones that use exponential utility functions [Howard and Matheson, 1972] [Coraluppi and Marcus, 1999] and expected value-minus-variance-criterion [Heger, 1994], we do not need the transition probabilities, since we use a model-free approach. Moreover, we are also able to vary the daring factor during the life of the agent, since the learnt Q values do not depend on this factor.…”

Section: Discussionmentioning

confidence: 99%

“…There are several risk-sensitive approaches, such as the ones introduced by Howard and Matheson [Howard and Matheson, 1972] and Coraluppi and Marcus [Coraluppi and Marcus, 1999]. Both proposals make use of exponential utility functions.…”

Section: Related Workmentioning

confidence: 99%

Learning to Avoid Risky Actions

Malfáz

Salichs

2011

Cybernetics and Systems

View full text Add to dashboard Cite

When a reinforcement learning agent executes actions that can cause frequently damages to itself, it can learn, by using Q-learning, that these actions must not be executed again. However, there are other actions that do not cause damage frecuently, only once in a while: risky actions, such as parachuting. These actions may imply a big punishment to the agent and, depending on its personality, it would be better to avoid. Nevertheless, using the standard Q-learning algorithm the agent is not able to learn to avoid them, since the result of these actions can be positive in average. In this paper, an additional mechanism to Q-learning, inspired by the emotion of fear, is introduced in order to deal with those risky actions by considering the worst results of them. Moreover, there is a daring factor for adjusting the consideration of the risk. This mechanism is implemented on an autonomous agent living in a virtual environment. The results present the performance of the agent with different daring degrees.

show abstract

“…The ETC and the EDC were first studied in, e.g., [18,20,31,32,33] for the finite state space model. Results in [18,20,31,33], characterizing the (exponential) optimal value function and policies are extended in Chapter 3, to infinite state space models.…”

Section: Summary Of Resultsmentioning

confidence: 99%

“…Results in [18,20,31,33], characterizing the (exponential) optimal value function and policies are extended in Chapter 3, to infinite state space models. Moreover, optimization with respect to the set of randomized, history de pendent policies is considered here, whereas only Markovian deterministic policies were considered ia [18,20,31,32,33]; see also [7] for related results. In Section 3.1, we present the exponential version of the policy evaluation algorithm (over a finite horizon), which plays a key role in the rest of the chapter.…”

Section: Summary Of Resultsmentioning

confidence: 99%

Controlled Markov chains with exponential risk-sensitive criteria: modularity, structured policies and applications

Avila-Godoy

Fernández-Gaucherand²

Proceedings of the 37th IEEE Conference on Decision and Control (Cat. No.98CH36171)

View full text Add to dashboard Cite

In the unlikely event that the author dkj not send UMI a complete manuscript and there are missing pages, these will be noted. Also, if unauthorized copyright material had to be removed, a note will indicate the deletkm. DOCTOR OF PHILOSOPHY STATEMENT BY AUTHORThis dissertation has been submitted in partial fulfillment of requirements for an advanced degree at The University of Arizona and is deposited in the University Library to be made available to borrowers under rules of the Library.Brief quotations from this dissertation are allowable without special permission, provided that accurate acknowledgment of source is made. Requests for permission for extended quotation from or reproduction of this manuscript in whole or in part may be granted by the head of the major department or the Dean of the Graduate College when in his or her judgment the proposed use of the material is in the interests of scholarship. In all other instances, however, permission must be obtained from the author. Finally, I want to express my deepest thanks to Agustm, my husband, for the long hours of discussion we shared during my doctoral program, for his helpful comments and suggestions, and most of all for his love. I will be forever grateful for the gift of his presence in my life. DEDICATIONA mi madre, In^ Godoy de Avila, con mucho carino.A mis hijos, Agiistm, Ernesto y Julio, con todo mi amor.A1 companero de mi vida, Agiistm, por todo lo que ha significado para mi. en mi formacion profesional.A Jesus y Lupita, por la solidaridad que me brindaron en mementos diffciles durante mi programa de doctorado.A todos mis hermanos. The optimal control problem for CMC's with a countable state space, and with a general action space, is studied for (exponential) total and discounted risk-sensitive cost criteria. General (dynamic programming) results for the finite and the infinite horizon cases are obtained. A set of general conditions is presented to obtain stmctural properties of the optimal value function and policies. In particular, monotonicity properties of value functions and optimal policies are established. The approach fol lowed is to show the (sub)modularity of certain fimctions (related to the optimality equations). Four applications studies are used to illustrate the general results obtained is this dissertation: equipment replacement, optimal resource allocation, scheduling of uncertain jobs, and inventory control. TABLE OF CONTENTS

show abstract

Risk-Sensitive Markov Decision Processes

Cited by 432 publications

References 1 publication

Risk management policies for dynamic capacity control

Risk management policies for dynamic capacity control

Learning to Avoid Risky Actions

Controlled Markov chains with exponential risk-sensitive criteria: modularity, structured policies and applications

Contact Info

Product

Resources

About