Posterior Predictive Outlier Detection Using Sample Reweighting

Abstract

To facilitate checking and improvement of a Bayesian model, we define an outlier as an observation or group of observations that is "surprising" relative to its predictive distribution, under the model, given the remainder of the data. Hence outlyingness can be measured by the posterior predictive case-deleted p-value of any interesting scalar summary of the (possibly multivariate) observation. It is also sometimes useful to condition on a part of the data for the potentially outlying case, such as the pattern of missing data, thus defining a conditional outlier.When parameters of interest have been sampled from their posterior distribution, the case-deleted p-value can be calculated by reweighting the sample to reflect deletion of the target observation and then drawing from the predictive distribution of the observation, facilitating huge computational savings. One efficient extension to the basic reweighting approach exploits the conditional independence structure of most hierarchical models. Another extension uses weighted systematic sampling to accommodate the dependence structure of the MCMC sample.Outlier checks are illustrated in hierarchical models for two datasets, a standard linear hierarchical model for rat growth and a complex ordinal model for survey data with nonignorably missing responses.Sample code and data for the rat growth example are available as an on-line supplement.

Bayesian hierarchical modeling of substate area estimates from the Medicare CAHPS survey

“…We instead use importance weighting to compute cross‐validated predictive distributions, using techniques motivated by Bayesian outlier detection methods . Importance sampling allows us to reweight our posterior draws of θ to calculate p ( y d | y (− d ) ), the leave‐one‐out cross‐validated predictive distribution for y d given the data y (− d ) with observation y d left out, as shown in Appendix .…”

Flexible epidemiological model for estimates and short-term projections in generalised HIV/AIDS epidemics

“…We further evaluated the spline-based model with posterior predictive checks of in-sample fit by comparing posterior predicted prevalence to observed prevalence for each site-year observation. We performed these posterior predictive checks after constructing case-deleted posteriors for each ANC site's random effect via sample reweighting, 34 and we obtained predicted random effects for ANC prevalence via rejection sampling. 32 To assess the model's performance in out-of-sample projection, we truncated the last 3 years of data, re-fit the model and determined if the 95% credible intervals for prevalence covered the median projection from the model fit to the full ANC time series.…”