The Logical Consistency of Simultaneous Agnostic Hypothesis Tests

Abstract

Simultaneous hypothesis tests can fail to provide results that meet logical requirements. For example, if A and B are two statements such that A implies B, there exist tests that, based on the same data, reject B but not A. Such outcomes are generally inconvenient to statisticians (who want to communicate the results to practitioners in a simple fashion) and non-statisticians (confused by conflicting pieces of information). Based on this inconvenience, one might want to use tests that satisfy logical requirements. However, Izbicki and Esteves shows that the only tests that are in accordance with three logical requirements (monotonicity, invertibility and consonance) are trivial tests based on point estimation, which generally lack statistical optimality. As a possible solution to this dilemma, this paper adapts the above logical requirements to agnostic tests, in which one can accept, reject or remain agnostic with respect to a given hypothesis. Each of the logical requirements is characterized in terms of a Bayesian decision theoretic perspective. Contrary to the results obtained for regular hypothesis tests, there exist agnostic tests that satisfy all logical requirements and also perform well statistically. In particular, agnostic tests that fulfill all logical requirements are characterized as region estimator-based tests. Examples of such tests are provided.

Pragmatic Hypotheses in the Evolution of Science

“…For each hypothesis, an agnostic hypothesis test can either reject it (1), accept it (0), or remain agnostic (1/2) [34]. Esteves et al [2] shows that an agnostic hypothesis test is logically coherent if and only if it is based on a region estimator. Such tests are presented in Definition 7 and illustrated in Figure 3.…”

“…In order to overcome such an impossibility result, Esteves et al [2] propose agnostic hypothesis tests, which have three possible outputs: (A) accept the hypothesis, say H, (E) reject H, or (Y) remain agnostic about H. These tests can be made logically coherent while preserving desirable statistical properties. For instance, both conditions are satisfied by the Generalized Full Bayesian Significance Test (GFBST).…”

Logically-consistent hypothesis testing and the hexagon of oppositions

“…A statistical hypothesis test is a function that attributes credal modalities to each statistical hypothesis. Esteves et al (2016) introduces four logical consistency conditions for statistical hypothesis tests: invertibility, monotonicity, union consonance and intersection consonance. In the following, these properties are represented using geometric solids composed of hexagons of oppositions.…”

“…Izbicki and Esteves (2015) shows that a standard test is logically consistent if and only if it is based on point estimation, which generally does not satisfy statistical optimality. This grim result motivated Esteves et al (2016) to investigate agnostic hypothesis tests. Such a test is a function, L, that assigns to each hypothesis, H, a value in {0, 0.5, 1}.…”

“…Using this framework, Esteves et al (2016) shows that the agnostic generalizations of some standard hypothesis tests can achieve logical consistency. Indeed, a hypothesis test is logically consistent if and only if it is based on a region estimator.…”