The Problem of Multiple Inference in Studies Designed to Generate Hypotheses

Abstract

Epidemiologic research often involves the simultaneous assessment of associations between many risk factors and several disease outcomes. In such situations, often designed to generate hypotheses, multiple univariate hypothesis-testing is not an appropriate basis for inference. The number of true positive associations in a collection of many associations can be estimated by comparing the observed distribution of p values for the positive associations to a theoretical uniform distribution, or to the observed distribution of negative associations, or to an empiric randomization distribution. None of these approaches, however, will distinguish the true from the false positive associations. Various criteria for selecting a subset of associations to report are considered by the authors, including Bonferoni adjustment of p values, splitting the sample for searching and testing, Bayesian inference, and decision theory. The authors prefer an approach in which all associations in the data are reported, whether significant or not, followed by a ranking in order of priority for investigation using empirical Bayes techniques. Methods are illustrated by application to preliminary data from a study aimed at identifying hitherto unsuspected occupational carcinogens.

Disease modifying therapies in multiple sclerosis: Subcommittee of the American Academy of Neurology and the MS Council for Clinical Practice Guidelines [RETIRED]

“…This is especially true when the study under consideration reports multiple between-group statistical comparisons, because multiple comparisons markedly inflate the actual Type I error rate and require a much more stringent statistical adjustment. [11][12][13][14][15] There is also concern about the Type II error rate of clinical trials (i.e., the so-called ␤ error), which reflects the likelihood of concluding incorrectly that a useful treatment is of no value. 16 For example, one recent trial 17 found that after 2 years of treatment, sustained disability progression was nonsignificantly reduced by 12%.…”

Prostate cancer and exposure to pesticides in agricultural settings

“…Some authors have argued, however, that the assumption of a universal null hypothesis is not tenable when many possible comparisons are carried out and, as a consequence, each association should be evaluated in relation to a separate null hypothesis, taking into account the validity of the study and prior evidences from laboratory and epidemiological research. [12][13][14][15][16][17][18][19][20][21][22][23] With reference to the excesses reported here, the available experimental and epidemiological data suggest that DDT may act as a multiple cancerogen 24 through different epigenetic mechanisms of action. [25][26][27] Limited evidence is available on the ability of dicofol to induce liver cancer in mouse.…”

Nutrient-Wide Association Studies

“…For GWAS as well as nongenetic studies, the most powerful use of the data appears to be to analyze the data as a single combined data set with appropriate adjustment of statistical significance. 5,6 Confirmation of findings in independent data sets is desirable, and sometimes these are available to authors, but often this is not feasible and is done by others. Fortunately, false-positive reports are ultimately corrected by the scientific process, although more careful accounting for multiple comparisons and caution in the interpretation of findings can make this more efficient.…”