Bayesian Best-Arm Identification for Selecting Influenza Mitigation Strategies

Abstract

Pandemic influenza has the epidemic potential to kill millions of people. While various preventive measures exist (i.a., vaccination and school closures), deciding on strategies that lead to their most effective and efficient use remains challenging. To this end, individual-based epidemiological models are essential to assist decision makers in determining the best strategy to curb epidemic spread. However, individual-based models are computationally intensive and it is therefore pivotal to identify the optimal strategy using a minimal amount of model evaluations. Additionally, as epidemiological modeling experiments need to be planned, a computational budget needs to be specified a priori. Consequently, we present a new sampling technique to optimize the evaluation of preventive strategies using fixed budget best-arm identification algorithms. We use epidemiological modeling theory to derive knowledge about the reward distribution which we exploit using Bayesian best-arm identification algorithms (i.e., Top-two Thompson sampling and BayesGap). We evaluate these algorithms in a realistic experimental setting and demonstrate that it is possible to identify the optimal strategy using only a limited number of model evaluations, i.e., 2-to-3 times faster compared to the uniform sampling method, the predominant technique used for epidemiological decision making in the literature. Finally, we contribute and evaluate a statistic for Top-two Thompson sampling to inform the decision makers about the confidence of an arm recommendation.

Multi-Agent Thompson Sampling for Bandit Applications with Sparse Neighbourhood Structures

“…MATS is a Bayesian method, which means that it can leverage prior knowledge about the data distribution. This property is highly beneficial in many practical applications, e.g., influenza mitigation 18,19 and wind farm control 4 .…”

“…Additionally, due to its Bayesian nature, problem-specific priors can be specified. We argue that this has strong relevance in many practical fields, such as advertisement selection 16 and influenza mitigation 18,19 .…”

Deep Reinforcement Learning for Large-Scale Epidemic Control

“…Finally, we present a novel approach to investigate how intervention policies can be improved by enabling collaboration between different geographic districts, by formulating the setting as a multiagent problem, and by solving it using deep reinforcement learning algorithms. Next to stateful reinforcement learning, the use of multi-armed bandits has recently been explored to assist decision makers to efficiently select the optimal prevention strategy [18,17].…”

“…To properly understand these dynamics, and to study emergency scenarios, epidemiological models are necessary. Such models enable us to make predictions and to study the effect of prevention strategies in simulation [18]. The development of prevention strategies, which need to fulfil distinct criteria (i.a., prevalence, mortality, morbidity, cost), remains a challenging process.…”

Interactive Multi-objective Reinforcement Learning in Multi-armed Bandits with Gaussian Process Utility Models

“…may want to minimise response time, while also minimising fuel cost and the stress levels of the drivers. For example, an agent learning the best strategy to control epidemics may want to minimise number of infections and minimise the burden on society [13]. When the user is unable to accurately and a priori specify preferences with respect to such objectives for all hypothetically possible trade-offs, the user needs to be informed about the values of actually available trade-offs between objectives in order to make a well-informed decision.…”