Cross-Screening in Observational Studies That Test Many Hypotheses

Abstract

We discuss observational studies that test many causal hypotheses, either hypotheses about many outcomes or many treatments. To be credible an observational study that tests many causal hypotheses must demonstrate that its conclusions are neither artifacts of multiple testing nor of small biases from nonrandom treatment assignment. In a sense that needs to be defined carefully, hidden within a sensitivity analysis for nonrandom assignment is an enormous correction for multiple testing: in the absence of bias, it is extremely improbable that multiple testing alone would create an association insensitive to moderate biases. We propose a new strategy called "cross-screening," different from but motivated by recent work of Bogomolov and Heller on replicability. Cross-screening splits the data in half at random, uses the first half to plan a study carried out on the second half, then uses the second half to plan a study carried out on the first half, and reports the more favorable conclusions of the two studies correcting using the Bonferroni inequality for having done two studies. If the two studies happen to concur, then they achieve Bogomolov-Heller replicability; however, importantly, replicability is not required for strong control of the family-wise error rate, and either study alone suffices for firm conclusions. In randomized studies with a few hypotheses, cross-split screening is not an attractive method when compared with conventional methods of multiplicity control, but it can become attractive when hundreds or thousands of hypotheses are subjected to sensitivity analyses in an observational study. We illustrate the technique by comparing 46 biomarkers in individuals who consume large quantities of fish versus little or no fish.

Defining Multimorbidity in Older Surgical Patients

“…Step 2: Cross-screening of comorbidity sets for elevated risk: If 50,183 true null hypotheses are tested at the 0.05 level, it is expected 0.05×50,183 = 2,905 of them will be falsely rejected. To avoid identifying 2,905 nonsense comorbidity sets by chance, we used the cross-screening methodology of Zhao, Small, and Rosenbaum, 44 an alternative to the Bonferroni adjustment, which would require comorbidity sets to be significant at 0.05/50,183. In cross-screening, the data are split in half at random.…”

“…For detailed discussion of cross-screening, its properties and limitations, see Zhao, Small, and Rosenbaum. 44 Cross-screening strongly controls the family-wise error rate: the chance that it falsely rejects one or more true null hypotheses is at most 0.05, despite having screened 50,183 combinations of comorbidities. It is very unlikely that even one of our comorbidity sets fails to achieve its stated property of double the conditional odds of death relative to the reference population.…”

Sensitivity Analysis for Inverse Probability Weighting Estimators via the Percentile Bootstrap

“…This data set was also analysed by Zhao et al . () and is publicly available in the R package CrossScreening on the Comprehensive R Archive Network.…”

“…We used the 234 matched pairs that were created by Zhao et al . () as the basis of the sensitivity analysis. As mentioned previously, Rosenbaum's sensitivity analysis assumes a constant treatment effect to construct confidence intervals.…”

On Sensitivity Value of Pair-Matched Observational Studies

“…In this case, the p-value upper bound p Γ is conservative if Γ > 1, and an unnecessary price of multiplicity is paid in multiple comparisons. Based on this observation, Heller et al (2009) proposed a sample splitting procedure to screen out uninteresting outcomes and gain power; see also Zhao et al (2017). (2) The observational study may simply be a preliminary study.…”