Generalizing Across Multi-Objective Reward Functions in Deep Reinforcement Learning

Abstract

Many reinforcement-learning researchers treat the reward function as a part of the environment, meaning that the agent can only know the reward of a state if it encounters that state in a trial run. However, we argue that this is an unnecessary limitation and instead, the reward function should be provided to the learning algorithm. The advantage is that the algorithm can then use the reward function to check the reward for states that the agent hasn't even encountered yet. In addition, the algorithm can simultaneously learn policies for multiple reward functions. For each state, the algorithm would calculate the reward using each of the reward functions and add the rewards to its experience replay dataset. The Hindsight Experience Replay algorithm developed by Andrychowicz et al. (2017) does just this, and learns to generalize across a distribution of sparse, goal-based rewards. We extend this algorithm to linearly-weighted, multi-objective rewards and learn a single policy that can generalize across all linear combinations of the multi-objective reward. Whereas other multi-objective algorithms teach the Q-function to generalize across the reward weights, our algorithm enables the policy to generalize, and can thus be used with continuous actions.

Multi-Objective reward generalization: Improving performance of Deep Reinforcement Learning for selected applications in stock and cryptocurrency trading

“…We use a generalized, intrinsically Multi-Objective RL strategy for stock and cryptocurrency trading. We implement this by considering extensions of Multi-Objective Deep Q-Learning RL algorithm with experience replay and target network stabilization given in [6], and deploying it on the Nifty50 stock index and BTCUSD trading pair.…”

“…To the very best of our knowledge, ours is first application of Multi-Reward RL in the sense of [6] to financial data.…”

“…We consider variations of the Deep-Q-Learning algorithm with Hindsight Experience Replay and target network stabilization [14] (DQN-HER) for both standard Single-reward or Multi-reward structure (in the sense of [6]) applied to single asset datasets.…”

Multi-objective multi-agent decision making: a utility-based analysis and survey

“…In the last couple of years, interest in Deep MORL has intensified, although primarily in single-agent settings (see e.g. [1,33,47,70,74,82,106,111,112]). Very recently, single-objective multi-agent RL has received considerable attention as well [30,32,39,55,97,81,109,130].…”

Using Logical Specifications of Objectives in Multi-Objective Reinforcement Learning

“…For our MORL agent we implement a vanilla DQN as described by Mnih et al (2015), although our method is easily applicable to most RL algorithms. Recent advances in MTRL, such as the use of UVFAs for generalizing across goals, has become more common in multiple objective settings (Friedman & Fontaine, 2018;Abels et al, 2018). We also utilize UVFAs to generalize across As the agent gains experience, tuples of state, action, next state, terminal, and reward vector (s, a, s , t, r) are stored in a replay buffer for future training.…”