Generalized Thompson sampling for sequential decision-making and causal inference

Abstract

Purpose: Sampling an action according to the probability that the action is believed to be the optimal one is sometimes called Thompson sampling. Methods: Although mostly applied to bandit problems, Thompson sampling can also be used to solve sequential adaptive control problems, when the optimal policy is known for each possible environment. The predictive distribution over actions can then be constructed by a Bayesian superposition of the policies weighted by their posterior probability of being optimal. Results: Here we discuss two important features of this approach. First, we show in how far such generalized Thompson sampling can be regarded as an optimal strategy under limited information processing capabilities that constrain the sampling complexity of the decision-making process. Second, we show how such Thompson sampling can be extended to solve causal inference problems when interacting with an environment in a sequential fashion. Conclusion: In summary, our results suggest that Thompson sampling might not merely be a useful heuristic, but a principled method to address problems of adaptive sequential decision-making and causal inference.

Personalizing persuasive technologies: Explicit and implicit personalization using persuasion profiles

“…Formally, the personalization problem can be regarded a contextual bandit problem (Yue et al, 2012;Ortega and Braun, 2013). The context X is given by the individual identifier and presents itself to the system.…”

Information-Theoretic Bounded Rationality and ε-Optimality

“…This interpretation builds on previous work that has related computational and physical processes; see for example [54] for an overview. As discussed in the Methods, the cost of changing distributions can also be expressed in terms of complexity of sampling processes [50,51].…”

“…The computational complexity of the information-theoretic model of bounded rational decision making can also be interpreted in terms of a sampling complexity [50,51]. In particular, Equation (4) can be interpreted under a rejection sampling scheme where we want to obtain samples from P (a i ), but we are only able to sample from the distribution P 0 (a i ).…”

Expanding the Active Inference Landscape: More Intrinsic Motivations in the Perception-Action Loop

“…Another option discussed in the literature is Thompson sampling (Ortega and Braun, 2010, 2014; Aslanides et al, 2017). In our framework this corresponds to a two step action selection procedure where we first sample an environment and parameter pair $({\overline{ê}}_{t-1},\overline{\theta })$ from a posterior factor (Bayesian or variational)…”