Optimal Screening for Hepatocellular Carcinoma: A Restless Bandit Model

Abstract

This paper seeks an efficient way to screen a population of patients at risk for hepatocellular carcinoma when (1) each patient’s disease evolves stochastically and (2) there are limited screening resources shared by the population. Recent medical discoveries have shown that biological information can be learned at each screening to differentiate patients into varying levels of risk for cancer. We investigate how to exploit this knowledge to choose which patients to screen to maximize early-stage cancer detections while limiting resource usage. We model the problem as a family of restless bandits, with each patient’s disease progression evolving as a partially observable Markov decision process. We derive an optimal policy for this problem and discuss managerial insights into what characterizes more effective screening. To provide numerical evidence, we use two independent data sets of over 800 patients, one to train the optimal policy, and the other to build a computer simulation to act as a test bed for said policy. We are able to show that our policy detects 22% more early-stage cancers than current practice, while using the same amount of resource expenditure. We provide insights into the structure underlying our policy and discuss the implications of our findings. The e-companion is available at https://doi.org/10.1287/msom.2017.0697 .

Learn to Intervene: An Adaptive Learning Policy for Restless Bandits in Application to Preventive Healthcare

“…Bhattacharya 2018 models the problem of maximizing the coverage and spread of health information as an RMAB problem and proposes a hierarchical policy. Lee et al 2019 study the problem of screening patients to maximize early-stage cancer detection under limited resource, by formulating it as a subclass of RMAB. Similarly, Glazebrook et al 2006, Hsu 2018, Sombabu et al 2020, Liu and Zhao 2010 give Whittle indexability results for different subclasses of (hidden) Markov bandits.…”

“…A relevant model for this setting is restless multi-armed bandits (RMABs). RMABs with prior knowledge of uncertainty model have been studied for health interventions [Lee et al, 2019;Mate et al, 2020;Mate et al, 2021;Bhattacharya, 2018], sensor monitoring tasks [Iannello et al, 2012;Glazebrook et al, 2006], anti-poaching patrols [Qian et al, 2016], and uplift modeling in eCommerce platforms [Gubela et al, 2019]. Due to the unpredictability of human beneficiaries, it is unrealistic to know the uncertainty model a priori.…”

An asymptotically optimal heuristic for general nonstationary finite-horizon restless multi-armed, multi-action bandits

“…The RMABP is a general modeling framework encompassing applications in sequential selection of clinical trials in medicine, sensor management, manufacturing systems, queueing networks, appointment scheduling, and capacity management in healthcare. We refer interested readers to the classic text by Gittins et al (2011) and to Bertsimas and Nino-Mora (2000) (and the references therein) for a more detailed discussion of applications of RMABPs in clinical trials, manufacturing systems, and queueing networks, among others; to Washburn (2008), Mahajan and Teneketzis (2008), and Ahmad et al (2009) for applications in sensor management; and to Deo et al (2013), Ayer et al (2016), and Lee et al (2018) for applications in healthcare. The optimal policy for an RMABP is rarely an index policy and it is frequently difficult to determine in any tractable manner (cf.…”

Operationalization of Utilitarian and Egalitarian Objectives for Optimal Allocation of Health Care Resources

“…Policies for such resource allocation decisions are usually guided by populationlevel objectives. These objectives are particularly significant in emergencies and humanitarian health care, where the goal is primarily to increase the populationlevel health rather than that of any particular individual (Blanchet et al, 2013;Lee, Lavieri, & Volk, 2019). The two most common objectives are utilitarianism, which maximizes the aggregate health outcome of a population, and egalitarianism, which, for example, seeks to minimize health differences by maximizing the welfare of those who are worst off (Rawls, 1971).…”

“…Cevik et al (2018) present a constrained POMDP model to study the optimal allocation of limited mammography resources to screen a population. Lee et al (2019) optimize the use of limited resources for the screening of a population for hepatocellular carcinoma by modeling the problem as a family of restless bandits in which each patient's disease progression is assumed to evolve as a POMDP. Güneş, Örmeci, and Kunduzcu (2015) present a compartmental model for allocating limited colonoscopy resources between screening and diagnostic services, whereas Deo, Rajaram, Rath, Karmarkar, and Goetz (2015) deploy a compartmental model to plan for HIV screening, testing, and care.…”