Nonstochastic Bandits and Experts with Arm-Dependent Delays

Abstract

We study nonstochastic bandits and experts in a delayed setting where delays depend on both time and arms. While the setting in which delays only depend on time has been extensively studied, the arm-dependent delay setting better captures real-world applications at the cost of introducing new technical challenges. In the full information (experts) setting, we design an algorithm with a firstorder regret bound that reveals an interesting trade-off between delays and losses. We prove a similar first-order regret bound also for the bandit setting, when the learner is allowed to observe how many losses are missing. These are the first bounds in the delayed setting that depend on the losses and delays of the best arm only. When in the bandit setting no information other than the losses is observed, we still manage to prove a regret bound through a modification to the algorithm of Zimmert and Seldin (2020). Our analyses hinge on a novel bound on the drift, measuring how much better an algorithm can perform when given a look-ahead of one round.

Near-Optimal Regret for Adversarial MDP with Delayed Bandit Feedback

“…Delays in multi-arm bandit (MAB). Delays were extensively studied in MAB and optimization both in the stochastic setting (Agarwal & Duchi, 2012;Vernade et al, 2017;Pike-Burke et al, 2018;Cesa-Bianchi et al, 2018;Zhou et al, 2019;Gael et al, 2020;Lancewicki et al, 2021;Cohen et al, 2021a), and the adversarial setting (Quanrud & Khashabi, 2015;Cesa-Bianchi et al, 2016;Thune et al, 2019;Bistritz et al, 2019;Zimmert & Seldin, 2020;Ito et al, 2020;Gyorgy & Joulani, 2021;van der Hoeven & Cesa-Bianchi, 2021).…”