Being Optimistic to Be Conservative: Quickly Learning a CVaR Policy

Abstract

While maximizing expected return is the goal in most reinforcement learning approaches, risk-sensitive objectives such as conditional value at risk (CVaR) are more suitable for many high-stakes applications. However, relatively little is known about how to explore to quickly learn policies with good CVaR. In this paper, we present the first algorithm for sample-efficient learning of CVaR-optimal policies in Markov decision processes based on the optimism in the face of uncertainty principle. This method relies on a novel optimistic version of the distributional Bellman operator that moves probability mass from the lower to the upper tail of the return distribution. We prove asymptotic convergence and optimism of this operator for the tabular policy evaluation case. We further demonstrate that our algorithm finds CVaRoptimal policies substantially faster than existing baselines in several simulated environments with discrete and continuous state spaces.

Lyapunov Barrier Policy Optimization

“…Dalal et al (2018) et al, 2000) is commonly used in quantitative finance, which aims to maximize returns in the worst α% of cases. This allows the agent to ensure that it learns safe policies for deployment that achieve high reward under the aleatoric uncertainty of the MDP (Tang et al, 2019;Keramati et al, 2019;Tamar et al, 2014;Kalashnikov et al, 2018;Borkar & Jain, 2010;Chow & Ghavamzadeh, 2014).…”

Cautious Reinforcement Learning via Distributional Risk in the Dual Domain

“…An additional approach is Bayesian and distributional RL (Bellemare et al, 2017), which seeks to track a full posterior over returns. These approaches benefit from the fact that with access to a full distribution, one may define risk specifically, with, e.g., conditional value at risk (CVaR) (Keramati et al, 2019). One limitation is that succinctly parameterizing the value distribution intersects with approximate Bayesian computation, an active area of research (Yang et al, 2019).…”