Investigating the replicability of preclinical cancer biology

Abstract

Replicability is an important feature of scientific research, but aspects of contemporary research culture, such as an emphasis on novelty, can make replicability seem less important than it should be. The Reproducibility Project: Cancer Biology was set up to provide evidence about the replicability of preclinical research in cancer biology by repeating selected experiments from high-impact papers. A total of 50 experiments from 23 papers were repeated, generating data about the replicability of a total of 158 effects. Most of the original effects were positive effects (136), with the rest being null effects (22). A majority of the original effect sizes were reported as numerical values (117), with the rest being reported as representative images (41). We employed seven methods to assess replicability, and some of these methods were not suitable for all the effects in our sample. One method compared effect sizes: for positive effects, the median effect size in the replications was 85% smaller than the median effect size in the original experiments, and 92% of replication effect sizes were smaller than the original. The other methods were binary – the replication was either a success or a failure – and five of these methods could be used to assess both positive and null effects when effect sizes were reported as numerical values. For positive effects, 40% of replications (39/97) succeeded according to three or more of these five methods, and for null effects 80% of replications (12/15) were successful on this basis; combining positive and null effects, the success rate was 46% (51/112). A successful replication does not definitively confirm an original finding or its theoretical interpretation. Equally, a failure to replicate does not disconfirm a finding, but it does suggest that additional investigation is needed to establish its reliability.

Challenges for assessing replicability in preclinical cancer biology

“…However, the present evidence suggests that we should be concerned. As reported in Errington et al, 2021b , replication efforts frequently produced evidence that was weaker or inconsistent with original studies. These results corroborate similar efforts by pharmaceutical companies to replicate findings in cancer biology ( Begley and Ellis, 2012 ; Prinz et al, 2011 ), efforts by a non-profit biotech to replicate findings of potential drugs in a mouse model of amyotrophic lateral sclerosis ( Perrin, 2014 ) , and systematic replication efforts in other disciplines ( Camerer et al, 2016 ; Camerer et al, 2018 ; Cova et al, 2018 ; Ebersole et al, 2016 ; Ebersole et al, 2019 ; Klein et al, 2014 ; Klein et al, 2018 ; Open Science Collaboration, 2015 ; Steward et al, 2012 ).…”

“…Much of the concern about replicability in science is whether reported findings are credible ( Begley and Ellis, 2012 ; Camerer et al, 2016 ; Camerer et al, 2018 ; Errington et al, 2021b ; Open Science Collaboration, 2015 ; Prinz et al, 2011 ). Our experience conducting this project identifies a much more basic and fundamental concern about replication – it is hard to assess whether reported findings are credible.…”

“…The initial goal was to repeat 193 experiments from 53 high-impact papers published between 2010 and 2012, but the obstacles we encountered at every phase of the research lifecycle meant that we were only able to repeat 50 experiments from 23 papers. In a separate paper we report a meta-analysis of the results of those 50 experiments ( Errington et al, 2021b ). In this paper, we describe the challenges we confronted during the different phases of the research lifecycle.…”

Is preclinical research in cancer biology reproducible enough?

“…It might be tempting to view the 33 abandoned efforts as uninformative, but we may be able to actually learn from them because it is likely that the 17 published studies are biased in favor of work that is reproducible. Laboratories that were more confident about the reproducibility of their original publications, or more fastidious with their record keeping, might be more likely to cooperate with the RPCB than laboratories that were more doubtful or less fastidious (although the RPCB team did not observe a relationship between material sharing and replication rates; Errington et al, 2021b ).…”

“…By the end of the project 17 Replication Studies had been published; Replication Studies were not published for 12 of Registered Reports due to technical and/or budgetary problems, but the results from eight partially completed replications have been published ( Errington et al, 2021a ; Pelech et al, 2021 ). The RPCB project has also published a paper containing a meta-analysis of all the replications ( Errington et al, 2021b ) and, separately, a paper that describes how the project was carried out and some of the challenges encountered during it ( Errington et al, 2021c ). The meta-analysis covers a total of 158 different effects from 50 experiments in 23 papers.…”