Reinforcement learning with dynamic convex risk measures

Abstract

We develop an approach for solving time‐consistent risk‐sensitive stochastic optimization problems using model‐free reinforcement learning (RL). Specifically, we assume agents assess the risk of a sequence of random variables using dynamic convex risk measures. We employ a time‐consistent dynamic programming principle to determine the value of a particular policy, and develop policy gradient update rules that aid in obtaining optimal policies. We further develop an actor–critic style algorithm using neural networks to optimize over policies. Finally, we demonstrate the performance and flexibility of our approach by applying it to three optimization problems: statistical arbitrage trading strategies, financial hedging, and obstacle avoidance robot control.

Recent advances in reinforcement learning in finance

“…Financial applications such as statistical arbitrage and portfolio optimization are discussed with detailed numerical examples. Coache and Jaimungal (2021) develops a framework combining policy‐gradient‐based RL method and dynamic convex risk measures for solving time‐consistent risk‐sensitive stochastic optimization problems. However, there is no sample complexity or asymptotic convergence studied for the proposed algorithms in Jaimungal et al.…”

“…However, there is no sample complexity or asymptotic convergence studied for the proposed algorithms in Jaimungal et al. (2021); Coache and Jaimungal (2021).…”

Conditionally Elicitable Dynamic Risk Measures for Deep Reinforcement Learning

“…A recent development in risk-aware RL is that in [9]. The authors use dynamic convex risk measures and devise a model-free approach to solve finite-horizon RL problems in a time-consistent manner.…”

“…This extends the work from [63] that studies optimal stationary policies under dynamic coherent risk measures. The authors of [9] also demonstrate the performance and flexibility of their approach on several benchmark examples, which, by generating strategies that mitigate risk and not simply maximizing expectation, effectively accounts for uncertainty in the data-generating processes. In both works, one downside of the proposed actor-critic algorithms is the use of a nested simulation or simulation upon simulation approach.…”

“…Instead of assuming that the one-step conditional risk measures \rho t are convex (see, e.g., [9]) or coherent (see, e.g., [53,63]), we impose stronger properties to focus on a narrower class of risk measures so that we can develop more efficient learning methodologies that do not require nested simulations. In what follows, we assume that the one-step conditional risk measures \rho t are static spectral risk measures as in Definition 2.3.…”