Null Hypothesis Testing ≠ Scientific Inference: A Critique of the Shaky Premise at the Heart of the Science and Values Debate, and a Defense of Value‐Neutral Risk Assessment

Abstract

Many philosophers and statisticians argue that risk assessors are morally obligated to evaluate the probabilities and consequences of methodological error, and to base their decisions of whether to adopt a given parameter value, model, or hypothesis on those considerations. This argument is couched within the rubric of null hypothesis testing, which I suggest is a poor descriptive and normative model for risk assessment. Risk regulation is not primarily concerned with evaluating the probability of data conditional upon the null hypothesis, but rather with measuring risks, estimating the consequences of available courses of action and inaction, formally characterizing uncertainty, and deciding what to do based upon explicit values and decision criteria. In turn, I defend an ideal of value‐neutrality, whereby the core inferential tasks of risk assessment—such as weighing evidence, estimating parameters, and model selection—should be guided by the aim of correspondence to reality. This is not to say that value judgments be damned, but rather that they should be accounted for within a structured approach to decision analysis, rather than embedded within risk assessment in an informal manner.

Inductive Risk, Science, and Values: A Reply to MacGillivray

“…MacGillivray argues that risk assessment should aim to be value‐free in its “core scientific inference”—namely, “the analysis, synthesis, and interpretation of evidence.” This core, he writes, is “making and communicating informative, good predictions.” By informative he means that reports should be “relevant to some real world decision problem.” By good , he means “reliable” and aimed to “correspond” to the world (MacGillivray, , p. 1521). He does not remark that these two imperatives, relevance and reliability, are inevitably in tension.…”

“…Following the way that the early debate was posed by Rudner (), the AIR is sometimes presumed to apply only to the unqualified confirmation and acceptance of hypotheses. So critics such as Betz (), Jeffrey (), Mitchell (), and MacGillivray () argue that the problem can be evaded by not fully accepting any conclusion but instead just reporting probabilities, degrees of uncertainty, or other qualified claims. The AIR is broader than this, however—a point we return to below.…”

“…Some authors writing about the AIR take it to be about induction in one of these narrower senses. The title of MacGillivray () (“Null Hypothesis Testing ≠ Scientific Inference”) suggests that the scope of the paper's argument takes AIR to be essentially concerned with null hypothesis testing (NHST).…”

“…MacGillivray admits that that the AIR “putatively applies” to these decisions, but objects that it is better to explicitly use formal approaches to uncertainty analysis than to rely on implicit judgments of value. MacGillivray writes: “The general point is that there are formal methods for uncertainty analysis, which to a large extent remove the obligation of the analyst to estimate, through untutored introspection, the probabilities, and consequences of methodological error” (MacGillivray, , p. 1526). This point applies to the AIR understood narrowly, but it misses the mark against the broader argument.…”

“…MacGillivray () argues that risk analysts should hew to a value‐free ideal of science. In order to defend this ideal, he attempts to undercut an argument that has great currency among philosophers of science—the argument from inductive risk (AIR).…”