Batch-Size Independent Regret Bounds for the Combinatorial Multi-Armed Bandit Problem

Abstract

We consider the combinatorial multi-armed bandit (CMAB) problem, where the reward function is nonlinear. In this setting, the agent chooses a batch of arms on each round and receives feedback from each arm of the batch. The reward that the agent aims to maximize is a function of the selected arms and their expectations. In many applications, the reward function is highly nonlinear, and the performance of existing algorithms relies on a global Lipschitz constant to encapsulate the function's nonlinearity. This may lead to loose regret bounds, since by itself, a large gradient does not necessarily cause a large regret, but only in regions where the uncertainty in the reward's parameters is high. To overcome this problem, we introduce a new smoothness criterion, which we term Gini-weighted smoothness, that takes into account both the nonlinearity of the reward and concentration properties of the arms. We show that a linear dependence of the regret in the batch size in existing algorithms can be replaced by this smoothness parameter. This, in turn, leads to much tighter regret bounds when the smoothness parameter is batch-size independent. For example, in the probabilistic maximum coverage (PMC) problem, that has many applications, including influence maximization, diverse recommendations and more, we achieve dramatic improvements in the upper bounds. We also prove matching lower bounds for the PMC problem and show that our algorithm is tight, up to a logarithmic factor in the problem's parameters.

(Locally) Differentially Private Combinatorial Semi-Bandits

“…Recently, there are interesting results under Gini-weighted smoothness assumptions (Merlis & Mannor, 2019;2020). Compared with general Lipschitz smoothness considered in this work, this is a more refined smoothness assumption, which leads to near optimal regret bounds with less dependence on the dimension K. Directly applying our algorithms to this setting will lead to an additional dependence on K. How to remove this additional price for privacy preserving, and how to prove the corresponding lower bounds, are interesting problems for future work.…”

Top-$k$ eXtreme Contextual Bandits with Arm Hierarchy

“…We will now review the combinatorial action space literature in multi-armed bandit (MAB) problems. Most of the work in this space deals with semi-bandit feedback (Chen et al, 2016;Combes et al, 2015;Kveton et al, 2015;Merlis and Mannor, 2019). This is also our feedback model, but we work in a contextual setting.…”

“…There is also work in the full-bandit feedback setting, where one gets to observe only one representative reward for the whole set of arms chosen. This body of literature can be divided into the adversarial setting (Merlis and Mannor, 2019;Cesa-Bianchi and Lugosi, 2012) and the stochastic setting Agarwal and Aggarwal, 2018;Lin et al, 2014;Rejwan and Mansour, 2020).…”

Adversarial Combinatorial Bandits with General Non-linear Reward Functions

“…Combinatorial bandit is a classical question in machine learning and has been extensively studied under the settings of stochastic semi-bandits (Chen et al, 2013;Combes et al, 2015;Kveton et al, 2015;Chen et al, 2016a;Merlis & Mannor, 2019;, stochastic bandits (Agarwal & Aggarwal, 2018;Rejwan & Mansour, 2020;Kuroki et al, 2020), and adversarial linear bandits (Cesa-Bianchi & Lugosi, 2012;Bubeck et al, 2012;Audibert et al, 2014;Combes et al, 2015). In the above-mentioned works, either the reward link function g(•) is linear, or the model is stochastic (stationary).…”