Multiplayer Bandits: A Trekking Approach

Abstract

We study stochastic multi-armed bandits with many players. The players do not know the number of players, cannot communicate with each other and if multiple players select a common arm they collide and none of them receive any reward. We consider the static scenario, where the number of players remains fixed, and the dynamic scenario, where the players enter and leave at any time. We provide algorithms based on a novel 'trekking approach' that guarantees constant regret for the static case and sub-linear regret for the dynamic case with high probability. The trekking approach eliminates the need to estimate the number of players resulting in fewer collisions and improved regret performance compared to the state-of-theart algorithms. We also develop an epoch-less algorithm that eliminates any requirement of time synchronization across the players provided each player can detect the presence of other players on an arm. We validate our theoretical guarantees using simulation based and real test-bed based experiments.

Distributed Learning in Ad-Hoc Networks: A Multi-player Multi-armed Bandit Framework

“…Our Trekking for Static Network (TSN) for static and its variant Trekking for Dynamic Network (TDN) in [4] overcomes the above drawbacks. As opposed to existing algorithms which separate estimation and orthogonalization tasks, we show that users can be settled on the top N channel without knowing N. Specifically, TSN and TDN are based on novel trekking approach where users operating on a channel always looks to operate on a vacant channel with better quality.…”

“…The epoch-free TDN algorithm avoids such regret and hence significantly outperforms others. Additional simulation results considering various scenarios for static and dynamic networks are given in [4].…”

“…From the architecture perspective, radio terminals may not need spectrum sensing unit while operating in unlicensed and shared spectrum which makes them area and power efficient. However, various works have shown that sensing in the unlicensed spectrum significantly reduces the number of collisions which in turn improves the utilization of the spectrum and hence the throughput [3,4]. From the learning perspective, the design of distributed algorithm and its analysis largely depends on the type of spectrum.…”

Non-Stationary Representation Learning in Sequential Linear Bandits

“…Related Work: As a well-known model to capture the exploration-exploitation dilemma in decision-making, multiarmed bandits have attracted extensive attention (see [13] for a survey). A variety of generalized bandit problems have been investigated, where the situations with non-stationary reward functions [14], restless arms [15], satisficing reward objectives [16], heavy-tailed reward distributions [17], risk-averse decision-makers [18], and multiple players [19] are considered. Distributed algorithms have also been proposed to tackle bandit problems (e.g., see [20]- [27]).…”

Representation Learning for Context-Dependent Decision-Making

“…The Upper Confidence Bound (UCB) algorithm and its variants have proven their strength in tackling multi-armed bandit problems (e.g., see [5], [6]). Various generalizations of the classical bandit problem have been studied, in which nonstationary reward functions [7], [8], restless arms [9], satisficing reward objectives [10], risk-averse decision-makers [11], heavy-tailed reward distributions [12], and multiple players [13] are considered. Recently, increasing attention has been also paid to tackling bandit problems in a distributed fashion (e.g., see [14]- [17]).…”