Bootstrap Thompson Sampling and Sequential Decision Problems in the Behavioral Sciences

Eckles, Dean; Kaptein, Maurits

doi:10.1177/2158244019851675

Cited by 20 publications

(10 citation statements)

References 53 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…One perspective in the literature is that Thompson sampling, and its variants, describe a way of making decisions given uncertainty and that the use of a posterior for uncertainty quantification can be swapped with some alternative. One option is to use boostrap samples rather than posterior samples [Eckles andKaptein, 2019, Russo et al, 2018]. This has been used most substantially in reinforcement learning [Riquelme et al, 2018, Osband et al, 2016.…”

Section: Policy Learningmentioning

confidence: 99%

Adaptivity and Confounding in Multi-Armed Bandit Experiments

Qin¹,

Russo²

2022

Preprint

View full text Add to dashboard Cite

We explore a new model of bandit experiments where a potentially nonstationary sequence of contexts influences arms' performance. Context-unaware algorithms risk confounding while those that perform correct inference face information delays. Our main insight is that an algorithm we call deconfounted Thompson sampling strikes a delicate balance between adaptivity and robustness. Its adaptivity leads to optimal efficiency properties in easy stationary instances, but it displays surprising resilience in hard nonstationary ones which cause other adaptive algorithms to fail.

show abstract

Section: Policy Learningmentioning

confidence: 99%

Adaptivity and Confounding in Multi-Armed Bandit Experiments

Qin¹,

Russo²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Motivated by the above, and inspired by the existing relationship between bootstrap distributions Rubin (1981) and Bayesian posteriors (i.e., bootstrap distributions can be used to approximate posteriors; Efron, 2012;Newton and Raftery, 1994), Eckles and Kaptein (2019) formulated a Bootstrap Thompson Sampling (BTS) technique for replacing the posterior by an online bootstrap distribution of the point estimate μt at each time t. In empirical evaluations, authors showed that, in comparison with LinTS and other methods, BTS it is more robust to model misspecifications, thanks to the robustness of the bootstrap approach, and it can be easily adapted to dependent observations, a common feature of behavioral sciences Kaptein, 2014, 2019).…”

Section: Contextual Bandits With Lints Explorationmentioning

confidence: 99%

Reinforcement Learning in Modern Biostatistics: Constructing Optimal Adaptive Interventions

Deliu¹,

Williams²,

Chakraborty³

2022

Preprint

View full text Add to dashboard Cite

Reinforcement learning (RL) is acquiring a key role in the space of adaptive interventions (AIs), attracting a substantial interest within methodological and theoretical literature and becoming increasingly popular within health sciences. Despite potential benefits, its real-life application is still limited due to several operational and statistical challenges-in addition to ethical and cost issues among others-that remain open in part due to a poor communication and synergy between methodological and applied scientists. In this work, we aim to bridge the different domains that contribute to and may benefit from RL, under a unique framework that intersects the areas of RL, causal inference and AIs, among others. We provide the first unified instructive survey on RL methods for building AIs, encompassing both dynamic treatment regimes (DTRs) and just-in-time adaptive interventions in mobile health (mHealth). We outline similarities and differences between the two areas, and discuss their implications for using RL. We combine our relevant methodological knowledge with motivating studies in both DTRs and mHealth to illustrate the tremendous collaboration opportunities between statistical, RL and healthcare researchers in the space of AIs.

show abstract

“…In this case a bootstrap distribution over means can be used to approximate a posterior distribution [11,23]. Eckles et al [9,10] use a bootstrap distribution to replace the posterior distribution used in Thompson Sampling. This method is known as Bootstrap Thompson Sampling (BTS) [9] and was proposed in the multi-arm bandit setting.…”

Section: (Bootstrap) Thompson Samplingmentioning

confidence: 99%

Risk Aware and Multi-Objective Decision Making with Distributional Monte Carlo Tree Search

Hayes,

Reymond,

Roijers

et al. 2021

Preprint

View full text Add to dashboard Cite

In many risk-aware and multi-objective reinforcement learning settings, the utility of the user is derived from the single execution of a policy. In these settings, making decisions based on the average future returns is not suitable. For example, in a medical setting a patient may only have one opportunity to treat their illness. When making a decision, just the expected return -known in reinforcement learning as the value -cannot account for the potential range of adverse or positive outcomes a decision may have. Our key insight is that we should use the distribution over expected future returns differently to represent the critical information that the agent requires at decision time. In this paper, we propose Distributional Monte Carlo Tree Search, an algorithm that learns a posterior distribution over the utility of the different possible returns attainable from individual policy executions, resulting in good policies for both risk-aware and multi-objective settings. Moreover, our algorithm outperforms the state-of-the-art in multi-objective reinforcement learning for the expected utility of the returns.

show abstract

Bootstrap Thompson Sampling and Sequential Decision Problems in the Behavioral Sciences

Cited by 20 publications

References 53 publications

Adaptivity and Confounding in Multi-Armed Bandit Experiments

Adaptivity and Confounding in Multi-Armed Bandit Experiments

Reinforcement Learning in Modern Biostatistics: Constructing Optimal Adaptive Interventions

Risk Aware and Multi-Objective Decision Making with Distributional Monte Carlo Tree Search

Contact Info

Product

Resources

About