Contradictions in Highly Cited Clinical Research

Abstract

it is not uncommon for highly cited studies of clinical interventions and their outcomes to be contradicted by subsequent research. The author noted several possible factors for this finding, including study-to-study variability and publication bias.An additional contributor to the observed discrepancy among study results concerns the meaning of the typical marker of research significance, the P value. The P value is the probability of a finding, assuming that there is in truth no effect; it is not simply the probability that there is no effect. As noted by Browner and Newman, 2 research studies are analogous to diagnostic tests and P values are like falsepositive rates when what we really seek is akin to a positive predictive value.A diagnostic test with high sensitivity and specificity may still be ineffective if disease prevalence is low. Similarly, a small P value can be misleading if the prior probability of an effect is low. For example, application of Bayes' theorem shows that if the prior probability of an effect is 10%, a P value of .05 from a study with 80% power results in a "falsepositive report probability" 3 of 36%: this leads to a far different interpretation of the study findings than the P value alone would suggest.Because much high-profile research is notable for its surprise factor, the prior probabilities of true effect may often be relatively low. We should not then be surprised when subsequent studies provide contradictory results. As noted by Davidoff, 4 one solution to this quandary is to better understand Bayesian approaches to study analysis and interpretation. Until such approaches are rendered comprehensible to the average researcher, 5,6 however, we should at least understand the limitations of the P value. Financial Disclosures: None reported. 1. Ioannidis JPA. Contradicted and initially stronger effects in highly cited clinical research. JAMA. 2005;294:218-228. 2. Browner WS, Newman TB. Are all significant P values created equal? the analogy between diagnostic tests and clinical research. JAMA. 1987;257:2459-2463. 3. Wacholder S, Chanock S, Garcia-Closas M, et al. Assessing the probability that a positive report is false: an approach for molecular epidemiology studies.

Verification of the Chromosome Region 9q21 Association with Pelvic Organ Prolapse Using RegulomeDB Annotations

“…It is accepted that replication studies do not require genome-wide multiple testing correction [ 38 ]. In our study, we also did not adjust for multiple comparisons since we tested robust statistical associations [ 39 , 40 ]. The Bonferroni correction was applied when testing for SNP interaction.…”