Quantifying Observed Prior Impact

Abstract

We distinguish two questions (i) how much information does the prior contain? and (ii) what is the effect of the prior? Several measures have been proposed for quantifying effective prior sample size, for example Clarke [1996] and Morita et al. [2008]. However, these measures typically ignore the likelihood for the inference currently at hand, and therefore address (i) rather than (ii). Since in practice (ii) is of great concern, Reimherr et al. [2014] introduced a new class of effective prior sample size measures based on prior-likelihood discordance. We take this idea further towards its natural Bayesian conclusion by proposing measures of effective prior sample size that not only incorporate the general mathematical form of the likelihood but also the specific data at hand. Thus, our measures do not average across datasets from the working model, but condition on the current observed data. Consequently, our measures can be highly variable, but we demonstrate that this is because the impact of a prior can be highly variable. Our measures are Bayes estimates of meaningful quantities and well communicate the extent to which inference is determined by the prior, or framed differently, the amount of effort saved due to having prior information. We illustrate our ideas through a number of examples including a Gaussian conjugate model (continuous observations), a Beta-Binomial model (discrete observations), and a linear regression model (two unknown parameters). Future work on further developments of the methodology and an application to astronomy are discussed at the end.

The Wasserstein Impact Measure (WIM): a generally applicable, practical tool for quantifying prior impact in Bayesian statistics

“…The relation between the MOPESS and the WIM is not monotone in some panels due to the known high variability of the MOPESS (Jones et al, 2020), which speaks in favour of the WIM. One can notice the higher MOPESS values compared to the WIM, in particular for θ = 1 there is more variability in the MOPESS.…”

“…Therefore the concept of effective prior sample size (EPSS) has been introduced. We will follow here the definition of Reimherr et al (2014) of a new class of effective prior sample size measures based on prior-likelihood discordance, which is also referred to as (the degree of) prior-likelihood conflict in Jones et al (2020). Reimherr et al (2014) published the first algorithm on how to adjust for prior-likelihood conflict in the calculation of EPSS to answer the question: how many extra observations are needed to transform a posterior based on a baseline prior into the posterior based on the prior of interest?…”

“…The EPSS can also be negative, indicating that the baseline prior is in fact more impactful than the prior of interest. Jones et al (2020) have extended the method of Reimherr et al (2014) to a more general setting and their method also works for lower sample sizes (where the impact of the prior is particularly important). They calculate the mean observed prior effective sample size (MOPESS) according to the following steps:…”

“…A further disadvantage of the MOPESS appears in models with covariates such as in regression settings, because it requires assumptions to be made on the distribution of each covariate. Jones et al (2020) mention this issue, and also show through a simple linear regression example that the MOPESS in principle should work in higher dimensions, though this extension has not been touched upon outside of this one example.…”

“…Consequently, different measures of prior impact have been proposed over the years. A popular approach is the so-called effective sample size, defined as the approximate number of observations equivalent to the information conveyed by the prior, see for instance Lin et al (2007); Morita et al (2008); Reimherr et al (2014); Wiesenfarth and Calderazzo (2019); Jones et al (2020) and the references therein. Kerman (2011) introduced the Neutrality, corresponding to the posterior's tail probability to the left of the frequentist maximum likelihood estimate.…”