Measuring statistical evidence and multiple testing

Abstract

The measurement of statistical evidence is of considerable current interest in fields where statistical criteria are used to determine knowledge. The most commonly used approach to measuring such evidence is through the use of p-values, even though these are known to possess a number of properties that lead to doubts concerning their validity as measures of evidence. It is less well known that there are alternatives with the desired properties of a measure of statistical evidence. The measure of evidence given by the relative belief ratio is employed in this paper. A relative belief multiple testing algorithm was developed to control for false positives and false negatives through bounds on the evidence determined by measures of bias. The relative belief multiple testing algorithm was shown to be consistent and to possess an optimal property when considering the testing of a hypothesis randomly chosen from the collection of considered hypotheses. The relative belief multiple testing algorithm was applied to the problem of inducing sparsity. Priors were chosen via elicitation, and sparsity was induced only when justified by the evidence and there was no dependence on any particular form of a prior for this purpose.

The Measurement of Statistical Evidence as the Basis for Statistical Reasoning

“…A better approach, in our view, is to identify the problems which can be considered central and, as discussed in the paper, when problems are encountered that lie outside this core consider compromises to the theory that retain as much of the ideal approach as possible. Certainly Cao et al (2015), and Evans and Tomal (2018) are in this spirit. Additional applications to more significant statistical problems than those discussed here can be found in the papers cited in this paper.…”

Measuring and Controlling Bias for Some Bayesian Inferences and the Relation to Frequentist Criteria

“…Suppose y ∼ N n (Xβ, σ 2 I), where X ∈ R n×k is of rank k, (β, σ 2 ) ∈ R k × (0, ∞) is unknown, our interest is in predicting a future value y new ∼ N(w t β, σ 2 ) for some fixed known w and, putting ν = 1/σ 2 , the conjugate prior β | ν ∼ N k (β 0 , ν −1 Σ 0 ) ν ∼ gamma rate (α 0 , η 0 ) is used. Specifying the hyperparameters (β 0 , Σ 0 , α 0 , η 0 ) can be carried out using elicitation as discussed in [37].…”

The Support Interval

“…This behavior of the relative surprise interval has led Evans (2015) to recommend reporting instead a so-called plausible interval, which only contains parameter values which have evidence in their favor-in other words, a support interval for k = 1 . These ideas have also been expanded upon by Baskurt and Evans (2013) in the context of evidence calibration, and by Evans and Tomal (2018) in the context of multiple testing and sparsity.…”