Softmax exploration strategies for multiobjective reinforcement learning

Abstract

Despite growing interest over recent years in applying reinforcement learning to multiobjective problems, there has been little research into the applicability and effectiveness of exploration strategies within the multiobjective context. This work considers several widely-used approaches to exploration from the single-objective reinforcement learning literature, and examines their incorporation into multiobjective Q-learning. In particular this paper proposes two novel approaches which extend the softmax operator to work with vectorvalued rewards. The performance of these exploration strategies is evaluated across a set of benchmark environments. Issues arising from the multiobjective formulation of these benchmarks which impact on the performance of the exploration strategies are identified. It is shown that of the techniques considered, the combination of the novel softmax-epsilon exploration with optimistic initialisation provides the most effective trade-off between exploration and exploitation.

Sparse Markov Decision Processes With Causal Sparse Tsallis Entropy Regularization for Reinforcement Learning

“…In order to address the issues with a deterministic policy function, a causal entropy regularization method has been utilized [6]- [10]. This is mainly due to the fact that the optimal solution of an MDP with causal entropy regularization becomes a softmax distribution of state-action values Q(s, a), i.e., π(a|s) = exp(Q(s,a)) a exp(Q(s,a )) , which is often referred to as a soft MDP [11].…”

Steering approaches to Pareto-optimal multiobjective reinforcement learning

“…The results in Section 7 indicated this did not occur when the target was balanced with regards to both objectives,which suggests the cause is that an unbalanced target leads to some base policies being active only infrequently, resulting in their learning being slowed due to more frequent clearing of the eligibility traces. Future work should investigate whether alternative approaches to exploration (such as optimistic initialisation, or exploratory selection at policy rather than action level) may ameliorate this issue [33].…”

Special issue on multi-objective reinforcement learning

“…The Deep Sea Treasure task [2,3,6] is a bi-objective environment consisting of ten Pareto-optimal states, which has often been used for testing MORL algorithms. The Bonus World used in [7] is an original three objective environment. Another bi-objective environment that has been used to evaluate a novel multi-objective RL algorithm is the Linked Rings problem [3].…”

“…The methodological approach. Many of the proposed MORL algorithms use variants of the Q-learning algorithm [2][3][4][5][6][7]. In [5], multi-objectivization is used to create additional objectives next to solving the primary goal in order to improve the empirical efficiency.…”

“…The secondary objectives are found using unsupervised learning and their corresponding learned policies are useful when the primary task changes in the environment. In [7], one objective is considered more important than the second objective with so-called lexicographic ordering, and to solve this problem the RL algorithm is integrated with new variants of the softmax exploration strategy. In another line of reasoning, in [2] the authors use Pareto dominance to partially order policies.…”