Learn to Intervene: An Adaptive Learning Policy for Restless Bandits in Application to Preventive Healthcare

Abstract

In many public health settings, it is important for patients to adhere to health programs, such as taking medications and periodic health checks. Unfortunately, beneficiaries may gradually disengage from such programs, which is detrimental to their health. A concrete example of gradual disengagement has been observed by an organization that carries out a free automated call-based program for spreading preventive care information among pregnant women. Many women stop picking up calls after being enrolled for a few months. To avoid such disengagements, it is important to provide timely interventions. Such interventions are often expensive and can be provided to only a small fraction of the beneficiaries. We model this scenario as a restless multi-armed bandit (RMAB) problem, where each beneficiary is assumed to transition from one state to another depending on the intervention. Moreover, since the transition probabilities are unknown a priori, we propose a Whittle index based Q-Learning mechanism and show that it converges to the optimal solution. Our method improves over existing learning-based methods for RMABs on multiple benchmarks from literature and also on the maternal healthcare dataset.

Whittle Index based Q-Learning for Wireless Edge Caching with Linear Function Approximation

“…We depart from [10] by studying the discounted counterpart as motivated by [42] since the difference in the optimal value between the discounted and average settings is small as long as α is close to 1 [40], [41]. Recently, another line of work [43] leveraged Q-learning to approximate Whittle indices through a single-timescale SA where Q-function and Whittle indices were learned independently. [43] considered the finite-horizon MDP and cannot be directly applied to infinite-horizon discounted or average reward MDPs.…”

“…Recently, another line of work [43] leveraged Q-learning to approximate Whittle indices through a single-timescale SA where Q-function and Whittle indices were learned independently. [43] considered the finite-horizon MDP and cannot be directly applied to infinite-horizon discounted or average reward MDPs. Finally, we are the first to provide a finite-time analysis of Whittle index based Q-learning, which further differentiates our work.…”

“…Q-learning based Algorithms. We compare our Q-Whittle learning to existing Q-learning algorithms (see Remark 3) when system parameters are unknown: (a) Q-learning Whittle Index Controller (Fu) [11]; (b) Q learning for Whittle index (AB) [10]; (c) Whittle Index Q-learning (WIQL) [43]; and (d) our Whittle policy, i.e., assume full knowledge of underlying transition probabilities. The discount factor is α = 0.8, learning rates are initialized to γ(0) = 0.1 and η(0) = 0.01, and are decayed by half every 1, 000 time steps.…”

Q-Learning Lagrange Policies for Multi-Action Restless Bandits

“…However, this is suboptimal for general RMABs since rewards are state-and action-dependent. Addressing this, Biswas et al [5] give a Q-learning-based based algorithm that acts on the arms that have the largest difference between their active and passive Q values. Fu et al [8] take a related approach that adjust the Q values by some 𝜆, and use it to estimate the Whittle index.…”

“…To address this shortcoming in previous work, this paper presents the first algorithms for the online setting for multi-action RMABs. Indeed, the online setting for even binary-action RMABs has received only limited attention, in the works of Fu et al [8], Avrachenkov and Borkar [3], and Biswas et al [5,6]. These papers adopt variants of the Q-learning update rule [29,30], a well studied reinforcement learning algorithm, for estimating the effect of each action across changing dynamics of the systems.…”