Statistically Efficient, Polynomial-Time Algorithms for Combinatorial Semi-Bandits

Abstract

We consider combinatorial semi-bandits over a set X ⊂ {0, 1} d where rewards are uncorrelated across items. For this problem, the algorithm ESCB yields the smallest known regret bound R(Tafter T rounds, where m = max x ∈X 1 ⊤ x. However, ESCB has computational complexity O(|X|), which is typically exponential in d, and cannot be used in large dimensions. We propose the first algorithm that is both computationally and statistically efficient for this problem with regret R(T, where δ T is a function which vanishes arbitrarily slowly. Our approach involves carefully designing AESCB, an approximate version of ESCB with the same regret guarantees. We show that, whenever budgeted linear maximization over X can be solved up to a given approximation ratio, AESCB is implementable in polynomial time O(δ −1 T poly(d )) by repeatedly maximizing a linear function over X subject to a linear budget constraint, and showing how to solve these maximization problems efficiently. Additional algorithms, proofs and numerical experiments are given in the complete version of this work.

Asymptotically Optimal Strategies For Combinatorial Semi-Bandits in Polynomial Time

“…We now highlight the combinatorial structures considered here, which include a large amount of classical and important structures for applications to real-world problems. We consider the same combinatorial structures as [7]. More on combinatorial structures and optimization can be found in [15] and references therein.…”

“…Approximate P BLM means that we can solve P BLM up to a given approximation ratio. The authors in [7] provide algorithms for the polynomial cases depicted in this table. Solving P BLM , either exactly or approximately, is the cornerstone of our approach to design asymptotically optimal algorithms.…”

“…ESCB [6] is an improved version of CUCB taking advantage of the fact that rewards are not correlated. AESCB [7] is an approximate version of ESCB with lower computational complexity. TS [19] is an algorithm inspired by Bayesian approaches.…”

“…Combinatorial semi-bandits have been widely studied. Many authors proposed algorithms and regret upper bounds, including Combinatorial Upper Confidence Bound (CUCB) [12], Efficient Sampling for Combinatorial Bandits (ESCB) [6,9], Approximate Efficient Sampling for Combinatorial Bandits (AESCB) [7] and a combinatorial version of Thompson sampling (TS) [19]. An information-theoretic regret lower bound was also provided by [6].…”

On the Suboptimality of Thompson Sampling in High Dimensions

“…ESCB [5,7] is an improvement of CUCB which leverages the independence of rewards between items. AESCB [6] is an approximate version of ESCB with roughly the same performance guarantees Algo. and reduced computational complexity.…”