Where Have All the Rodents Gone? The Effects of Attrition in Experimental Research on Cancer and Stroke

Holman, Cheryl; Piper, Sophie K.; Grittner, Ulrike; Diamantaras, Andreas Antonios; Kimmelman, Jonathan; Siegerink, Bob; Dirnagl, Ulrich

doi:10.1371/journal.pbio.1002331

Cited by 107 publications

(122 citation statements)

References 27 publications

Supporting

Mentioning

121

Contrasting

Unclassified

Order By: Relevance

“…A particularly problematic aspect of researcher flexibility is the decision to remove outliers after having seen their influence on the P-value. Selective removal of outliers has a high potential of generating biased results (Holman et al, 2016), so the removal of data points is generally discouraged. Publications should always explain the reasons behind any attrition (loss of data points between study initiation and data analysis) and should discuss whether the missed samples might have led to biased results.…”

Section: (E) Researcher Degrees Of Freedom: (2) Flexibility In Analysismentioning

confidence: 99%

Detecting and avoiding likely false‐positive findings – a practical guide

2016

View full text Add to dashboard Cite

Recently there has been a growing concern that many published research findings do not hold up in attempts to replicate them. We argue that this problem may originate from a culture of 'you can publish if you found a significant effect'. This culture creates a systematic bias against the null hypothesis which renders meta-analyses questionable and may even lead to a situation where hypotheses become difficult to falsify. In order to pinpoint the sources of error and possible solutions, we review current scientific practices with regard to their effect on the probability of drawing a false-positive conclusion. We explain why the proportion of published false-positive findings is expected to increase with (i) decreasing sample size, (ii) increasing pursuit of novelty, (iii) various forms of multiple testing and researcher flexibility, and (iv) incorrect P-values, especially due to unaccounted pseudoreplication, i.e. the non-independence of data points (clustered data). We provide examples showing how statistical pitfalls and psychological traps lead to conclusions that are biased and unreliable, and we show how these mistakes can be avoided. Ultimately, we hope to contribute to a culture of 'you can publish if your study is rigorous'. To this end, we highlight promising strategies towards making science more objective. Specifically, we enthusiastically encourage scientists to preregister their studies (including a priori hypotheses and complete analysis plans), to blind observers to treatment groups during data collection and analysis, and unconditionally to report all results. Also, we advocate reallocating some efforts away from seeking novelty and discovery and towards replicating important research findings of one's own and of others for the benefit of the scientific community as a whole. We believe these efforts will be aided by a shift in evaluation criteria away from the current system which values metrics of 'impact' almost exclusively and towards a system which explicitly values indices of scientific rigour.

show abstract

Section: (E) Researcher Degrees Of Freedom: (2) Flexibility In Analysismentioning

confidence: 99%

Detecting and avoiding likely false‐positive findings – a practical guide

2016

View full text Add to dashboard Cite

show abstract

“…Of considerable importance is the internal and external validity of this approach when broadened in its application [4, 71–73]. Poor internal validity has been identified as a major contributor to false positive results in pre-clinical studies [4, 5].…”

Section: A Proposal For a Pre-clinical Methodology That Incorporates mentioning

confidence: 99%

“…With numerous neuroprotectant compounds tested and failed [2, 3], the inability to predict successful clinical outcomes from apparently positive pre-clinical and early-phase clinical studies is all too common. While there are likely several explanations for these failures, including lack of power [4, 5], rigor in design and execution, and a disconnection between the pre-clinical design and human trial [6, 7], we believe there is a fundamental problem in the identification of effective agents due to the way clinical and pre-clinical studies are designed and analyzed for disorders heterogeneous with respect to the myriad of factors that influence outcome, including innate biological, environmental, and methodological variance and experimental noise [8]. …”

Section: Introductionmentioning

confidence: 99%

Embracing Biological and Methodological Variance in a New Approach to Pre-Clinical Stroke Testing

Kent

2016

Transl. Stroke Res.

View full text Add to dashboard Cite

High-profile failures in stroke clinical trials have discouraged clinical translation of neuroprotectants. While there are several plausible explanations for these failures, we believe that the fundamental problem is the way clinical and pre-clinical studies are designed and analyzed for heterogeneous disorders such as stroke due to innate biological and methodological variability that current methods cannot capture. Recent efforts to address pre-clinical rigor and design, while important, are unable to account for variability present even in genetically homogenous rodents. Indeed, efforts to minimize variability may lessen the clinical relevance of preclinical models. We propose a new approach that recognizes the important role of baseline stroke severity and other factors in influencing outcome. Analogous to clinical trials, we propose reporting baseline factors that influence outcome and then adapting for the pre-clinical setting a method developed for clinical trial analysis where the influence of baseline factors is mathematically modeled and the variance quantified. A new therapy’s effectiveness is then evaluated relative to the pooled outcome variance at its own baseline conditions. In this way, an objective threshold for robustness can be established that must be overcome to suggest its effectiveness when expanded to broader populations outside of the controlled environment of the PI’s laboratory. The method is model neutral and subsumes sources of variance as reflected in baseline factors such as initial stroke severity. We propose that this new approach deserves consideration for providing an objective method to select agents worthy of the commitment of time and resources in translation to clinical trials.

show abstract

“…Of those that did report numbers, ≈30% reported that they had dropped rodents from their study analysis, but <25% of those explained why. Using simulated data, we demonstrated that this can lead to a major distortion of the results, 20 especially when group sizes are small.…”

Section: Can Mice Mimic Human Strokementioning

confidence: 99%

Thomas Willis Lecture

Dirnagl

2016

Stroke

Self Cite

View full text Add to dashboard Cite

Abstract-Based on research, mainly in rodents, tremendous progress has been made in our basic understanding of the pathophysiology of stroke. After many failures, however, few scientists today deny that bench-to-bedside translation in stroke has a disappointing track record. I here summarize many measures to improve the predictiveness of preclinical stroke research, some of which are currently in various stages of implementation: We must reduce preventable (detrimental) attrition. Key measures for this revolve around improving preclinical study design. Internal validity must be improved by reducing bias; external validity will improve by including aged, comorbid rodents of both sexes in our modeling. False-positives and inflated effect sizes can be reduced by increasing statistical power, which necessitates increasing group sizes. Compliance to reporting guidelines and checklists needs to be enforced by journals and funders. Customizing study designs to exploratory and confirmatory studies will leverage the complementary strengths of both modes of investigation. All studies should publish their full data sets. On the other hand, we should embrace inevitable NULL results. This entails planning experiments in such a way that they produce high-quality evidence when NULL results are obtained and making these available to the community. A collaborative effort is needed to implement some of these recommendations. Just as in clinical medicine, multicenter approaches help to obtain sufficient group sizes and robust results. Translational stroke research is not broken, but its engine needs an overhauling to render more predictive results.

show abstract

Where Have All the Rodents Gone? The Effects of Attrition in Experimental Research on Cancer and Stroke

Cited by 107 publications

References 27 publications

Detecting and avoiding likely false‐positive findings – a practical guide

Detecting and avoiding likely false‐positive findings – a practical guide

Embracing Biological and Methodological Variance in a New Approach to Pre-Clinical Stroke Testing

Thomas Willis Lecture

Contact Info

Product

Resources

About