A survey of Bayesian predictive methods for model assessment, selection and comparison

Vehtari, Aki; Ojanen, Janne

doi:10.1214/12-ss102

Cited by 314 publications

(282 citation statements)

References 143 publications

Supporting

Mentioning

281

Contrasting

Order By: Relevance

“…Note that it is not necessarily the case that the more complex the model, the better the predictive performance: as shown later, a complex model may well have as good or poorer performance than a simpler model (Gelman, Hwang, & Vehtari, 2014;Vehtari et al, 2012). An alternative to using elpd is to examine −2 × elpd, which is equivalent to deviance, and is called the LOO Information Criterion.…”

Section: Model Comparisonmentioning

confidence: 96%

See 1 more Smart Citation

Bayesian Hierarchical Finite Mixture Models of Reading Times: A Case Study

Vasishth¹,

Nicenboim²,

Chopin³

et al. 2017

Preprint

View full text Add to dashboard Cite

This theoretical note presents a case study demonstrating the importance of Bayesian hierarchical mixture models as a modelling tool for evaluating the predictions of competing theories of cognitive processes. This note also contributes to improving current practices in data analysis in the psychological sciences. As a case study, we revisit two published data sets from psycholinguistics. In sentence comprehension, it is widely assumed that the distance between linguistic co-dependents affects the latency of dependency resolution: the longer the distance, the longer the time taken to complete the dependency (e.g., Gibson 2000). An alternative theory, direct access (McElree, 2000), assumes that retrieval times are a mixture of two distributions : one distribution represents successful retrievals and the other represents an initial failure to retrieve the correct dependent, followed by a reanalysis (McElree, 1993) that leads to successful retrieval. Here, dependency distance has the effect that in long-distance conditions the proportion of reanalyses is higher (due to similarity-based interference). We implement both theories as Bayesian hierarchical models and show that the direct-access model fits the Chinese relative clause reading time data better than the dependency-distance account. This work makes several novel contributions. First, we demonstrate how the researcher can reason about the underlying generative process of their data, thereby expressing the underlying cognitive process as a statistical model. Second, we show how models that have been developed in an exploratory manner to represent different underlying generative processes can be compared in terms of their predictive performance, using both K-fold cross validation on existing data, and using completely new data. Finally, we show how the models can be evaluated using simulated data; this is a method that is standardly used in Bayesian statistics, but remains unutilized in data analysis within the psychological sciences.

show abstract

Section: Model Comparisonmentioning

confidence: 96%

“…Bayesian model comparison can be carried out using different methods (see Vehtari, Ojanen, et al, 2012, for an extended discussion). Here, we use K-fold cross-validation (Vehtari, Gelman, & Gabry, 2017) because it is a well-known method for model evaluation, and because it is computationally tractable.…”

Section: Model Comparisonmentioning

confidence: 99%

Bayesian Hierarchical Finite Mixture Models of Reading Times: A Case Study

Vasishth¹,

Nicenboim²,

Chopin³

et al. 2017

Preprint

View full text Add to dashboard Cite

show abstract

“…Recent computational developments have made the computation of Bayes factors more tractable, especially for common scenarios (64,65). For uncommon or complex scenarios, one might resort to reporting a different model comparison metric that does not rely on the marginal likelihood, such as the various information criteria (AIC, BIC, DIC, WAIC) or leave-one-out cross validation (LOOCV; see 56,59,60). However, it should be emphasized that for the purposes of inference these alternative methods can be suboptimal.…”

Section: T Ementioning

confidence: 99%

Introduction to Bayesian Inference for Psychology

2017

View full text Add to dashboard Cite

“…Chaloner & Verdinelli 1995;Myung & Pitt 2009;Curtis 2004a,b;Maurer et al 2010) and theory of Bayesian or statistical inference (e.g. Tarantola 2005;Vehtari & Ojanen 2012) provide machinery which in theory deliver the optimal experimental design, and procedures for efficient estimation and inference. However in order for this machinery to function, the questioner must-explicitly or implicitly-specify models of the system under consideration, quantify prior knowledge, identify the space of possible experimental designs, and quantify in the form of a utility function the risks or rewards associated with drawing all possible conclusions in all possible worlds.…”

Section: Introductionmentioning

confidence: 99%

Interrogation theory

Arnold

Curtis

2018

Geophysical Journal International

View full text Add to dashboard Cite

A B S T R A C TThe goal of an investigation, scientific or otherwise, is usually to find answers to some specific set of questions about the state of nature: what is the seismic velocity structure? How likely is this volcano to erupt within a certain period? Does a subsurface reservoir contain resources of interest? Background research may reveal the existence of pertinent knowledge and information discovered previously, new data are normally acquired, and an inference problem is solved in order to answer the questions taking both all of the a priori information and the new data into account. Inverse theory, decision theory and the theory of experimental design provide methods to optimize the design of the investigation and to estimate results. However, those theories are normally set in the context of a particular model of the universe, with its particular parametrization. This requires the investigator to specify a priori a coherent utility (a function that describes the risks and rewards) of all possible outcomes under that parametrization. Quite commonly, the investigator may not be able to do this. Ideally an investigator would be able merely to pose a set of questions, define a set of constraints on the data types, acquisition costs and logistics, and provide a functional to relate the questions to any particular parameter space. Theory and methodology would then semi-autonomously drive the interrogation of the state of nature by optimally selecting one or more relevant models and parameter spaces, and designing, acquiring and analysing data, in order to best answer the questions. If necessary this could be done in a sequential or iterative manner, which potentially then involves changing the questions posed in each iteration based on both previous results and inspiration from the investigator. We present such a theory of interrogation in this paper. We review the relevant aspects of decision and design theory, and cast them in a framework where the investigator specifies a utility only at the level required by the general questions to be posed. Each model under consideration is then mapped into this utility space of possible answers. We then extend this framework to sequential investigations, where the outcome of each step may affect all aspects of the problem: the models entertained, the utilities and even the questions themselves. A variety of examples illustrates the generality of this method: an asset team investigating how best to exploit a subsurface reservoir, Monte Carlo sampling to estimate the Bayesian evidence for geophysical models, discriminating between different rock physics models of strain in laboratory deformation experiments, an organization sequentially assessing the effectiveness of its methods to evaluate subsurface assets, assessing whether subsurface CO 2 storage should be promoted for climate change mitigation, and examples running through the text of seismic tomography, earthquake characterization and autonomous interplanetary robotic exploration.

show abstract

A survey of Bayesian predictive methods for model assessment, selection and comparison

Cited by 314 publications

References 143 publications

Bayesian Hierarchical Finite Mixture Models of Reading Times: A Case Study

Bayesian Hierarchical Finite Mixture Models of Reading Times: A Case Study

Introduction to Bayesian Inference for Psychology

Interrogation theory

Contact Info

Product

Resources

About