A Learning Algorithm for Risk-Sensitive Cost

Abstract

A linear function approximation based reinforcement learning algorithm is proposed for Markov decision processes with infinite horizon risk-sensitive cost. Its convergence is proved using the 'o.d.e. method' for stochastic approximation. The scheme is also extended to continuous state space processes. 1. Introduction. Recent decades have seen a major activity in approximate dynamic programming for Markov decision processes based on real or simulated data, using reinforcement learning algorithms. (See, e.g., Bertsekas and Tsitsiklis (1996) [10] and Sutton and Barto (1998) [30] for book length treatments and Si et al (2004) [28] for a flavour of more recent activity. While most of this work has focused on the additive cost criteria such as discounted or time-averaged cost, relatively little has been done for the multiplicative cost (or risk-sensitive cost as it is better known). There is, however, a lot of interest in this cost criterion, as it has important applications in finance, e.g., Bagchi and Sureshkumar (2002) (1983) [5] were developed. These were 'raw' in the sense that there was no explicit approximation of the value function in order to beat down the curse of dimensionality. In case of additive costs, there is a considerable body of work on such approximation architectures, one of the most popular being the linear function approximation. Here one seeks an approximation of the value function in terms of linear combination of a moderate number of basis functions specified a priori. The learning scheme then iteratively learns the weights (or coefficients) of this linear combination instead of learning the full value function, which is a much higher dimensional object. The first rigorous analysis of such a scheme is in Tsitsiklis and Van Roy (1997) [31], where its convergence was proved for the problem of policy evaluation. Since then there have been several variations of the basic theme, see, e.g., Bertsekas, Borkar and Nedic (2004) [8] and the references therein. The aim of this article is to propose a similar linear function approximation based learning scheme for policy evaluation in risk-sensitive control and justify it rigorously.

Variance-constrained actor-critic algorithms for discounted and average reward MDPs

“…and we know that it is much easier to design actor-critic or other reinforcement learning algorithms (Borkar 2001(Borkar , 2002Basu et al 2008;Borkar 2010) for this risk measure than those that will be presented in this paper. However, this formulation is limited in the sense that it requires knowing the ideal tradeoff between the mean and variance, since it takes β as an input.…”

“…Most of the work on this topic (including those mentioned above) has been in the context of MDPs (when the model of the system is known) and much less work has been done within the reinforcement learning (RL) framework (when the model is unknown and all the information about the system is obtained from the samples resulted from the agent's interaction with the environment). In risk-sensitive RL, we can mention the work by Borkar (2001Borkar ( , 2002Borkar ( , 2010 and Basu et al (2008) who considered the expected exponential utility, the one by Mihatsch and Neuneier (2002) that formulated a new risk-sensitive control framework based on transforming the temporal difference errors that occur during learning, and the one by Tamar et al (2012) on several variance related measures. Tamar et al (2012) study stochastic shortest path problems, and in this context, propose a policy gradient algorithm [and in a more recent work (Tamar and Mannor 2013) an actor-critic algorithm] for maximizing several risk-sensitive criteria that involve both the expectation and variance of the return random variable (defined as the sum of the rewards that the agent obtains in an episode).…”

Risk-sensitive control for a class of nonlinear systems with multiplicative noise

“…The discrete-time partial observation problem was solved by Whittle in [33] (see also [34]). For infinite horizon criterion in a Markovian setting, the reader can consult [5], [9], [10]. An important relation with robust controllers was found in [14], [15], whereas the risk-sensitive maximum principle was studied in [26], [27], [17], [20].…”

Compound Reinforcement Learning: Theory and an Application to Finance

“…Their optimality, however, is based on the expected discounted rewards. In this paper, we focus on the compound return 1 . The aim of this research is to maximize the compound return by extending the RL framework.…”

“…Averagereward RL [6,12,13,15] maximizes the arithmetic average rewards in reward-based MDPs. Risk-sensitive RL [1,2,5,7,9,11] not only maximizes the sum of expected discounted rewards, it also minimizes the risk defined by each study. While they can learn risk-averse behavior, they do not take into account maximizing the compound return.…”