Multi-Dueling Bandits and Their Application to Online Ranker Evaluation

Abstract

New ranking algorithms are continually being developed and refined, necessitating the development of efficient methods for evaluating these rankers. Online ranker evaluation focuses on the challenge of efficiently determining, from implicit user feedback, which ranker out of a finite set of rankers is the best.Online ranker evaluation can be modeled by dueling bandits, a mathematical model for online learning under limited feedback from pairwise comparisons. Comparisons of pairs of rankers is performed by interleaving their result sets and examining which documents users click on. The dueling bandits model addresses the key issue of which pair of rankers to compare at each iteration, thereby providing a solution to the exploration-exploitation trade-off.Recently, methods for simultaneously comparing more than two rankers have been developed. However, the question of which rankers to compare at each iteration was left open. We address this question by proposing a generalization of the dueling bandits model that uses simultaneous comparisons of an unrestricted number of rankers.We evaluate our algorithm on synthetic data and several standard large-scale online ranker evaluation datasets. Our experimental results show that the algorithm yields orders of magnitude improvement in performance compared to stateof-the-art dueling bandit algorithms.

Dueling Bandits with Team Comparisons

“…Dueling bandits with sets of actions One line of dueling bandits extension consider the case where the learner selects a subset of actions and observes the outcomes of all duels between all pairs of actions in the subset [2,22], or the winner of the subset [20,19]. As a consequence, these settings give the learner strictly more information than the dueling bandits setting.…”

“…We refine this idea by a binary search approach, decreasing the number of duels to log(k). We remark that Lemma C.2 in the appendix is a slightly stronger version of the above lemma, which allows us to partition A and B into two subsets each, A = A (1) ∪ A (2) and B = B (1) ∪ B (2) . Under some circumstances, we can then guarantee that Uncover reveals the pairwise comparison between two players a ≻ b, where a is from A (1) and b is from B (1) .…”

“…Uncover Subroutine As sketched within the main part of our paper, we refine the idea of the Uncover subroutine by a binary search approach. Moreover, we add the option to input a refinement of A and B, namely A = A (1) ∪ A (2) , B = B (1) ∪ B (2) , guaranteeing that the uncovered relation is between a pair of players from A (1) and B (1) , while A (2) and B (2) are contained in one of the sets of the witness each. For that to work, we require that 2) , and 2) .…”

“…Input: four disjoint sets, A (1) , B (1) , A (2) , B (2) with |A (1) | = |B (1) |, |A (2) | = |B (2) |, |A (1) | + |A (2) | = k, A (1) ∪ A (2) ≻ B (1) ∪ B (2) , and A (1) ∪ B (2) ≻ B (1) ∪ A (2) Output: a ∈ A (1) , b ∈ B (1) , (S, S ′ ) ∈ S * a,b with (C ⊆ S and D ⊆ S ′ ) or (D ⊆ S and C ⊆ S ′ ) Set S ← A (1) ∪ A (2) , T ← B (1) ∪ B (2) , ℓ ← 1, r ← |A (1)…”

Efficient and Optimal Algorithms for Contextual Dueling Bandits under Realizability

“…Performance is often measured by some notion of regret, and while many definitions have been studied [Yue et al, 2012;Zoghi et al, 2014;, they all intuitively ask that the learner identify the good actions, i.e., those that are typically favored amongst the others. Over the last two decades, several algorithms have been proposed for dueling bandit problems [Ailon et al, 2014;Zoghi et al, 2014;Komiyama et al, 2015;Wu and Liu, 2016] and generalizations to subset-wise preference feedback [Sui et al, 2017;Brost et al, 2016;Saha and Gopalan, 2019a;Ren et al, 2018;Saha and Gopalan, 2019b].…”