Designing multi-objective multi-armed bandits algorithms: A study

Drugan, Mădălina M.; Nowé, Ann

doi:10.1109/ijcnn.2013.6707036

Cited by 88 publications

(79 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the multi-objective case, i.e., a multi-objective multi-armed bandit problem (MOBP) (Drugan and Nowé, 2013), we typically need more than one policy, and deterministic policies no longer suffice.…”

Section: Bandit Problemsmentioning

confidence: 99%

See 1 more Smart Citation

Multi-objective decision-theoretic planning

Roijers

2016

AI Matters

View full text Add to dashboard Cite

Decision making is hard. It o en requires reasoning about uncertain environments, partial observability and action spaces that are too large to enumerate. In such complex decisionmaking tasks decision-theoretic agents, that can reason about their environments on the basis of mathematical models and produce policies that optimize the utility for their users, can o en assist us.In most research on decision-theoretic agents, the desirability of actions and their e ects is codi ed in a scalar reward function. However, many real-world decision problems have multiple objectives. In such cases the problem is more naturally expressed using a vector-valued reward function. Rather than having a single optimal policy, we then want to produce a set of policies that covers all possible preferences between the objectives. We call such a set a coverage set. In this dissertation, we focus on decision-theoretic planning algorithms that produce the convex coverage set (CCS), which is the optimal solution set when either: 1) the user utility can be expressed as a weighted sum over the values for each objective; or 2) policies can be stochastic.We propose new methods based on two popular approaches to creating planning algorithms that produce an (approximate) CCS by building on an existing single-objective algorithm. In the inner loop approach, we replace the summations and maximizations in the inner most loops of the single-objective algorithm by cross-sums and pruning operations. In the outer loop approach, we solve a multi-objective problem as a series of scalarized problems by employing the single-objective method as a subroutine.Our most important contribution is an outer loop framework that we call optimistic linear support (OLS). As an outer loop method OLS builds the CCS incrementally. We show that, contrary to existing outer loop methods, each intermediate result is a bounded approximation of the CCS with known bounds (even when the single-objective method used is a bounded approximate method as well) and is guaranteed to terminate in a nite number of iterations.We apply OLS-based algorithms to a variety of multi-objective decision problems, and show that it is more memory-e cient, and faster than corresponding inner loop algorithms for moderate numbers of objectives. We show that exchanging subroutines in OLS is relatively easy and illustrate the importance on a complex planning problem. Finally, we show that it is o en possible to reuse parts of the policies and values, found in earlier iterations of OLS, to hot-start later iterations of OLS. Using this last insight, we propose the rst method for multi-objective POMDPs that employs point-based planning and can produce an ε-CCS in reasonable time.Overall, the methods we propose bring us closer to truly practical multi-objective decisiontheoretic planning.

show abstract

Section: Bandit Problemsmentioning

confidence: 99%

“…However, the learning setting is beyond the scope of this dissertation. Please refer to (Auer and Ortner, 2010;Kuleshov and Precup, 2014) for an overview of BP learning algorithms, and to (Drugan and Nowé, 2013;Yahyaa et al, 2014) for MOBP algorithms.…”

mentioning

confidence: 99%

Multi-objective decision-theoretic planning

Roijers

2016

AI Matters

View full text Add to dashboard Cite

show abstract

“…The multi-objective multi-armed bandits (MOMAB) [Drugan and Nowe, 2013] algorithms are MABs with reward vectors that import techniques from multi-objective optimisation for an efficient exploration / exploitation trade-off. There are important differences between the MOMAB and the standard MAB algorithms that arise mainly because: (1) there are sets of arms that can be considered to be the best, i.e.…”

Section: Introductionmentioning

confidence: 99%

Learning and Intelligent Optimization

Dhaenens

Jourdan

Marmion

2015

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. This paper presents an analysis of different possible operators for local search algorithms in order to solve permutation-based problems. These operators can be defined by a distance metric that define the neighborhood of the current configuration, and a selector that chooses the next configuration to be explored within this neighborhood. The performance of local search algorithms strongly depends on their ability to efficiently explore and exploit the search space. We propose here a methodological approach in order to study the properties of distances and selectors in order to buildtheir performances operators that can be used either for intensification of the search or for diversification stages. Based on different observations, this approach allows us to define a simple generic hyperheuristic that adapt the choice of its operators to the problem at hand and that manages their use in order to ensure a good trade-off between intensification and diversification. Moreover this hyperheuristic can be used on different permutation-based problems.

show abstract

“…The well known exploration/exploitation trade-off plays a very important role in this problem [11]. These methods have also used both the Pareto dominance relation [12] and the scalarization functions [13] to identify the Pareto front. Some multi-objective reinforcement learning algorithms use the lexicographical order relation [14] that assumes that one objective is more important than another objective.…”

Section: Introductionmentioning

confidence: 99%

Model-based multi-objective reinforcement learning

Wiering

Withagen

Drugan

2014

2014 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL)

View full text Add to dashboard Cite

Abstract-This paper describes a novel multi-objective reinforcement learning algorithm. The proposed algorithm first learns a model of the multi-objective sequential decision making problem, after which this learned model is used by a multiobjective dynamic programming method to compute Pareto optimal policies. The advantage of this model-based multi-objective reinforcement learning method is that once an accurate model has been estimated from the experiences of an agent in some environment, the dynamic programming method will compute all Pareto optimal policies. Therefore it is important that the agent explores the environment in an intelligent way by using a good exploration strategy. In this paper we have supplied the agent with two different exploration strategies and compare their effectiveness in estimating accurate models within a reasonable amount of time. The experimental results show that our method with the best exploration strategy is able to quickly learn all Pareto optimal policies for the Deep Sea Treasure problem. I. INTRODUCTION Reinforcement learning (RL) [1], [2]enables an autonomous agent to learn from its interactions with a particular environment that emits reward signals to the agent. The objective of the agent is to learn a policy that obtains the highest possible discounted cumulative reward intake. In this paper we consider value-based reinforcement learning, where the agent estimates a value function denoting the future reward intake and uses this value function to select actions. Many valuebased reinforcement learning algorithms have been proposed [2]. These algorithms can be divided into model-free and model-based reinforcement learning algorithms. Model-free methods such as Q-learning [3] update the Q-value function after each interaction with the environment without estimating a model. Model-based RL methods first learn to estimate a model of the environment and then use a dynamic programming algorithm to compute the policy. The advantage of model-based RL methods is that experiences of the agent are used more effectively, leading to faster convergence to optimal policies.Although traditionally reinforcement learning algorithms have been applied solely to single objective decision problems, during the last decade the amount of research on multiobjective problems has considerably increased [4], [5]. In multi-objective reinforcement learning (MORL), the reward function emits a reward vector instead of a single scalar reward, and the goal is to learn all Pareto optimal policies.

show abstract

Designing multi-objective multi-armed bandits algorithms: A study

Cited by 88 publications

References 7 publications

Multi-objective decision-theoretic planning

Multi-objective decision-theoretic planning

Learning and Intelligent Optimization

Model-based multi-objective reinforcement learning

Contact Info

Product

Resources

About