The non-Bayesian restless multi-armed bandit: A case of near-logarithmic regret

Abstract

In the classic Bayesian restless multi-armed bandit (RMAB) problem, there are N arms, with rewards on all arms evolving at each time as Markov chains with known parameters. A player seeks to activate K ≥ 1 arms at each time in order to maximize the expected total reward obtained over multiple plays. RMAB is a challenging problem that is known to be PSPACE-hard in general. We consider in this work the even harder non-Bayesian RMAB, in which the parameters of the Markov chain are assumed to be unknown a priori. We develop an original approach to this problem that is applicable when the corresponding Bayesian problem has the structure that, depending on the known parameter values, the optimal solution is one of a prescribed finite set of policies. In such settings, we propose to learn the optimal policy for the non-Bayesian RMAB by employing a suitable meta-policy which treats each policy from this finite set as an arm in a different non-Bayesian multi-armed bandit problem for which a single-arm selection policy is optimal. We demonstrate this approach by developing a novel sensing policy for opportunistic spectrum access over unknown dynamic channels. We prove that our policy achieves near-logarithmic regret (the difference in expected reward compared to a modelaware genie), which leads to the same average reward that can be achieved by the optimal policy under a known model. This is the first such result in the literature for a non-Bayesian RMAB. For our proof, we also develop a novel generalization of the Chernoff-Hoeffding bound.

Learning in A Changing World: Non-Bayesian Restless Multi-Armed Bandit

“…The policy proposed in [10] also uses the index form of UCB-1 given in [5], but the structure is different from RUCB proposed in this paper. In [11], a stronger definition of regret is adopted, where regret is defined as reward loss with respect to the optimal performance in the ideal scenario of known reward model. However, the problem can only be solved for a special class of RMAB.…”

“…Specifically, when arms are governed by stochastically identical two-state Markov chains, a policy was constructed in [11] to achieve a regret with an order arbitrarily close to logarithmic.…”

On a restless multi-armed bandit problem with non-identical arms

“…In [4], it has been shown that this structure can be exploited to obtain an efficient online learning algorithm for the non-Bayesian version of the problem (where the underlying transition matrix is completely unknown). In particular, Dai et al [4] show that near logarithmic regret (defined as the difference between cumulative rewards obtained by a model-aware optimal-policy-implementing genie and that obtained by their policy) with respect to time 1 can be achieved by mapping two particular policies to arms in a different multi-armed bandit. For the more general case of non-identical arms, there have been some recent results that show near-logarithmic weak regret (measured with respect to the best possible single-channel-selection policy, which need not be optimal) [5], [6], [9].…”

“…The problem of dynamic channel selection has recently been formulated and studied by many researchers [1], [2], [3], [4], [5], [6] under the framework of multi-armed bandits (MAB) [7]. In these papers, the channels are typically modelled as indpendent Gilbert-Elliott channels (i.e., described by two-state Markov chains, with a bad state "0" and a good state "1").…”

Decentralized learning for multi-player multi-armed bandits

“…[10] proposes a policy based on deterministic sequence of exploration and exploitation and achieves the same bounds for weak regret. In [11], the authors consider the notion of strong regret and propose a policy which achieves near-log T (strong) regret for some special cases of the restless model.…”