Admissibility in Partial Conjunction Testing

Abstract

Meta-analysis combines results from multiple studies aiming to increase power in finding their common effect. It would typically reject the null hypothesis of no effect if any one of the studies shows strong significance. The partial conjunction null hypothesis is rejected only when at least r of n component hypotheses are non-null with r = 1 corresponding to a usual metaanalysis. Compared with meta-analysis, it can encourage replicable findings across studies. A by-product of it when applied to different r values is a confidence interval of r quantifying the proportion of non-null studies. Benjamini and Heller (2008) provided a valid test for the partial conjunction null by ignoring the r − 1 smallest p-values and applying a valid meta-analysis p-value to the remaining n − r + 1 p-values. We provide sufficient and necessary conditions of admissible combined p-value for the partial conjunction hypothesis among monotone tests. Non-monotone tests always dominate monotone tests but are usually too unreasonable to be used in practice. Based on these findings, we propose a generalized form of Benjamini and Heller's test which allows usage of various types of meta-analysis p-values, and apply our method to an example in assessing replicable benefit of new anticoagulants across subgroups of patients for stroke prevention.

Replicability of Treatment Effect in Study of Blood Pressure Lowering With Dementia

“…If the r value is less than the type I error (eg, 0.05), it can be concluded that there was a treatment effect in at least 2 studies. Such a conclusion can be established even if none of the individual studies is statistically significant . If the r value is greater than 0.05, it does not preclude the effect from being true but emphasizes that the current evidence may rely critically on a single study.…”

“…We suggest enhancing the meta-analysis with a replicability analysis. Such analyses provide an r value that quantifies the evidence of replicability (ie, whether the effect is present in at least 2 studies) . If the r value is less than the type I error (eg, 0.05), it can be concluded that there was a treatment effect in at least 2 studies.…”