Risk-Sensitive Markov Control Processes

Abstract

We introduce the Lyapunov approach to optimal control problems of average risk-sensitive Markov control processes with general risk maps. Motivated by applications in particular to behavioral economics, we consider possibly nonconvex risk maps, modeling behavior with mixed risk preference. We introduce classical objective functions to the risk-sensitive setting and we are in particular interested in optimizing the average risk in the infinite-time horizon for Markov Control Processes on general, possibly non-compact, state spaces allowing also unbounded cost. Existence and uniqueness of an optimal control is obtained with a fixed point theorem applied to the nonlinear map modeling the risksensitive expected total cost. The necessary contraction is obtained in a suitable chosen seminorm under a new set of conditions: 1) Lyapunov-type conditions on both risk maps and cost functions that control the growth of iterations, and 2) Doeblin-type conditions, known for Markov chains, generalized to nonlinear mappings. In the particular case of the entropic risk map, the above conditions can be replaced by the existence of a Lyapunov function, a local Doeblin-type condition for the underlying Markov chain, and a growth condition on the cost functions.

Variance-constrained actor-critic algorithms for discounted and average reward MDPs

“…In risk-sensitive sequential decision-making, the objective is to maximize a risk-sensitive criterion such as the expected exponential utility (Howard and Matheson 1972), a variance related measure (Sobel 1982;Filar et al 1989), the percentile performance (Filar et al 1995), or conditional value-at-risk (CVaR) (Ruszczyński 2010;Shen et al 2013). Unfortunately, when we include a measure of risk in our optimality criteria, the corresponding optimal policy is usually no longer Markovian stationary (e.g., Filar et al 1989) and/or computing it is not tractable (e.g., Filar et al 1989;Mannor and Tsitsiklis 2011).…”

Risk-averse reinforcement learning for algorithmic trading

“…The risk-averse optimal trade strategy is then obtained by maximizing the following risk-averse objective [5]:…”

“…Hence, it cannot measure the risk associated with μ s,a efficiently. For more detailed discussion, readers may refer to [5].…”

“…In this paper, we propose a general framework of riskaverse trading algorithms based on the risk-sensitive Markov decision processes (RS-MDP, [5], [6]) to solve a common high-frequency trade problem faced by an institutional trader: the optimal trade execution (see e.g., [7], [8], [9] and references therein), i.e., to liquid a huge inventory over a short time horizon. Specifically, we derive a risk-averse objective function within the framework of RS-MDP, which balances the expected revenue and the associated risk by applying a nonlinear transformation.…”

“…The authors also did experiments with an additional feature of order book, imbalance of depths and conclude that its effects on the performance of all RL algorithms are marginal 5. Our model can be extent to include the order size as another dimension of the action space, given an estimate for the market impact of limit order (see e.g.,[19]).…”