On the Whittle Index for Restless Multiarmed Hidden Markov Bandits

Abstract

We consider a restless multi-armed bandit in which each arm can be in one of two states. When an arm is sampled, the state of the arm is not available to the sampler. Instead, a binary signal with a known randomness that depends on the state of the arm is available. No signal is available if the arm is not sampled. An arm-dependent reward is accrued from each sampling. In each time step, each arm changes state according to known transition probabilities which in turn depend on whether the arm is sampled or not sampled. Since the state of the arm is never visible and has to be inferred from the current belief and a possible binary signal, we call this the hidden Markov bandit. Our interest is in a policy to select the arm(s) in each time step that maximizes the infinite horizon discounted reward. Specifically, we seek the use of Whittle's index in selecting the arms.We first analyze the single-armed bandit and show that in general, it admits an approximate threshold-type optimal policy when there is a positive reward for the 'no-sample' action. We also identify several special cases for which the threshold policy is indeed the optimal policy. Next, we show that such a singlearmed bandit also satisfies an approximate-indexability property. For the case when the single-armed bandit admits a thresholdtype optimal policy, we perform the calculation of the Whittle index for each arm. Numerical examples illustrate the analytical results.

Learning in Restless Multi-Armed Bandits using Adaptive Arm Sequencing Rules

“…In [29], the MAB problem was defined with satisfying objectives so that the player aims at obtaining a reward above a certain threshold. In [30], a two states RMAB problem was defined in a hidden nature. Specifically, when an arm is sampled, the state of the arm is not fully observable.…”

Sequential Decision Making With Limited Observation Capability: Application to Wireless Networks

“…This additional information also makes computation of Whittle-index expressions easier. In recent work of [36], [37], hidden Markov restless multi-armed bandit has been studied and Whittle-index policy is used. This model assumes that arm state is never fully observable but only binary signals corresponding to each state transition are observed.…”

Rested and Restless Bandits With Constrained Arms and Hidden States: Applications in Social Networks and 5G Networks

“…3) When θ a = 0 or θ a = 1 for all a ∈ {0, 1}, our definitions of indexability and index are still valid. To claim indexability, we will require to show that π th (w) and π th (w) are non-increasing in w. Now, we use the following lemma from [10]. , then π th (w) and π th (w) are monotonically decreasing functions of w. Now, using Lemma 2 and Definition 2, we can show that single-armed restless bandit is indexable.…”

“…RMAB assumes that the model of system state variations is known. The Whittle index based policies also studied for opportunistic communication systems in [9], [10], where authors studied partially observable model. The Whittle index policies are popular due to they are asymptotically optimal.…”

“…optimal for certain model parameter. In [10], a hidden Markov RMAB is studied which generalizes the work of [9]. The myopic policy was also investigated for restless bandits in [11] and it is shown to be optimal under certain model assumptions.…”