Assessing the Global and Local Uncertainty of Scientific Evidence in the Presence of Model Misspecification

Abstract

Scientists need to compare the support for models based on observed phenomena. The main goal of the evidential paradigm is to quantify the strength of evidence in the data for a reference model relative to an alternative model. This is done via an evidence function, such as ΔSIC, an estimator of the sample size scaled difference of divergences between the generating mechanism and the competing models. To use evidence, either for decision making or as a guide to the accumulation of knowledge, an understanding of the uncertainty in the evidence is needed. This uncertainty is well characterized by the standard statistical theory of estimation. Unfortunately, the standard theory breaks down if the models are misspecified, as is commonly the case in scientific studies. We develop non-parametric bootstrap methodologies for estimating the sampling distribution of the evidence estimator under model misspecification. This sampling distribution allows us to determine how secure we are in our evidential statement. We characterize this uncertainty in the strength of evidence with two different types of confidence intervals, which we term “global” and “local.” We discuss how evidence uncertainty can be used to improve scientific inference and illustrate this with a reanalysis of the model identification problem in a prominent landscape ecology study using structural equations.

Ctenophores are direct developers that reproduce continuously beginning very early after hatching

“…Traditional generalized linear models to test the effect of different experimental and biological factors on the accumulation of births should not be used here because they erroneously ignore the biological time dependency in the counts and therefore may result in excessive type I errors in hypothesis testing, as well as severe model choice errors ( 33 ). The full hierarchical model derivation, our maximum likelihood parameter estimation, and our nonparametric bootstrap model selection approaches using evidential statistics and the Bayesian information criterion (BIC) ( 34 – 38 ) are detailed in the SI Appendix . SI Appendix , Fig.…”

“…To test the reliability and support in the resulting chosen best models for every experiment, we adopted the recent nonparametric bootstrap model selection methodology of Taper et al. ( 34 ). To do that, we performed sampling with replacement of each of the datasets while preserving the structure of the data, generated a set of 100 nonparametric bootstrap samples each time to generate nonparametric bootstrap ( n = 100) results, and compared the difference in the first two BICs (delta-ICs) under the best supported model vs. all other models (see SI Appendix , Data S1 for results of all models tested, including bootstrap results and first two delta-ICs for best model vs. all other models).…”

Entropy, Statistical Evidence, and Scientific Inference: Evidence Functions in Theory and Applications

“…(2) Examples where application of the LP leads to paradoxical results (e.g., Cornfield [ 15 ]). Additionally, (3) questions about the LP applicability to scientific inference [ 13 , 16 , 17 , 18 , 19 ], given that it was formulated and proved under a perfect model specification scenario, a criterion that is rarely met, if at all, in day-to-day scientific modeling settings.…”

“…Possible outcomes are strong evidence for model 1, weak and inconclusive evidence, and strong evidence for model 2. Taper et al [ 19 ] suggest further dividing the outcome continuum into strong evidence for model 1, prognostic evidence for model 1, weak evidence not really favoring either model, prognostic evidence for model 2, and strong evidence for model 2. Prognostic evidence is suggestive of a model, but not sufficiently strong to stand on its own without collateral evidence.…”

“…Since then, modern evidential statistics has extended the concept of evidence to statistical cases with unknown parameters and misspecified models, but it realizes that a frequentist error structure can be created for both pre-data design and post-data inference [ 15 , 19 , 23 , 30 ]. Further, both conditional uncertainty (i.e., uncertainty in evidence given the data in hand) and unconditional uncertainty (i.e., uncertainty in evidence over resampling of data) can be calculated [ 19 , 31 ].…”