A Survey of Preference-Based Online Learning with Bandit Algorithms

Abstract

Abstract. In machine learning, the notion of multi-armed bandits refers to a class of online learning problems, in which an agent is supposed to simultaneously explore and exploit a given set of choice alternatives in the course of a sequential decision process. In the standard setting, the agent learns from stochastic feedback in the form of real-valued rewards. In many applications, however, numerical reward signals are not readily available-instead, only weaker information is provided, in particular relative preferences in the form of qualitative comparisons between pairs of alternatives. This observation has motivated the study of variants of the multi-armed bandit problem, in which more general representations are used both for the type of feedback to learn from and the target of prediction. The aim of this paper is to provide a survey of the state-of-the-art in this field, that we refer to as preference-based multi-armed bandits. To this end, we provide an overview of problems that have been considered in the literature as well as methods for tackling them. Our systematization is mainly based on the assumptions made by these methods about the data-generating process and, related to this, the properties of the preference-based feedback.

Exploring embodiment and dueling bandit learning for preference adaptation in human-robot interaction

“…Hence, this work extends the literature by evaluating how reinforcement learning, in our case bandit learning, can be used to personalize the human's HRI experience. We therefore draw from research that extended multi-armed bandit learning scenario to a dueling bandit learning scenario [15]. In those scenarios the agent learns the user's preference by presenting the user two items.…”

“…Particularly, we study a special kind of bandit learning (i.e. dueling bandit learning [15]) for PL. In contrast to standard bandit learning techniques, this approach does not require a numerical reward function.…”

Preference-Based Monte Carlo Tree Search

“…Therefore we use RUCB in the following. For a general overview of PB-MAB algorithms, we refer the reader to [4].…”

Online Learning to Rank for Information Retrieval

“…• How to perform online LTR with a finite population of candidate rankers, framing it as a K-armed bandits problem [6].…”

“…• Online learning algorithms that have not been used in the LTR settings with the goal of inspiring researchers to adapt those algorithms for use in the IR community [4,5] [10 minutes] Datasets and resources.…”