Risk-Sensitive Policy with Distributional Reinforcement Learning

Abstract

Classical reinforcement learning (RL) techniques are generally concerned with the design of decision-making policies driven by the maximisation of the expected outcome. Nevertheless, this approach does not take into consideration the potential risk associated with the actions taken, which may be critical in certain applications. To address that issue, the present research work introduces a novel methodology based on distributional RL to derive sequential decision-making policies that are sensitive to the risk, the latter being modelled by the tail of the return probability distribution. The core idea is to replace the Q function generally standing at the core of learning schemes in RL by another function, taking into account both the expected return and the risk. Named the risk-based utility function U, it can be extracted from the random return distribution Z naturally learnt by any distributional RL algorithm. This enables the spanning of the complete potential trade-off between risk minimisation and expected return maximisation, in contrast to fully risk-averse methodologies. Fundamentally, this research yields a truly practical and accessible solution for learning risk-sensitive policies with minimal modification to the distributional RL algorithm, with an emphasis on the interpretability of the resulting decision-making process.

Offline reinforcement learning in high-dimensional stochastic environments

“…In risk-sensitive RL, the focus is on minimizing a measure of the risk induced by the cumulative rewards, rather the maximizing the expected return. Risk-sensitive RL has gained attention in recent years [40][41][42][43]. Various risk measures may be considered, such that Exponential Utility [44] Cumulative Prospect Theory [45] or Conditional Value-at-Risk (CVaR) [46].…”

Dopamine neurons encode a multidimensional probabilistic map of future reward

“…Such innovations have been shown to improve performance of deep RL agents on benchmark tasks due to improved statistical robustness, and evidence of distributional RL-like computations has been reported in midbrain DANs of both mice and primates 7,8,9 , however the direct functional relevance of such distributional mechanisms and representations to behavior is unknown. In the engineering setting, deep RL agents vary in whether and how they make use of knowledge about the distribution over future rewards when selecting actions [10][11][12] , and decoding of reward distributions from DAN activity has only been demonstrated at the time of reward delivery 7 . In principle, knowing in advance, at the start of an episode, about the range and likelihood of rewards available and when they are likely to occur could be highly useful for planning and flexible behavior, particularly in the face of non-stationary in either the environment or internal state (e.g.…”