Improving reproducibility in animal research by splitting the study population into several ‘mini-experiments’

Abstract

In light of the hotly discussed ‘reproducibility crisis’, a rethinking of current methodologies appears essential. Implementing multi-laboratory designs has been shown to enhance the external validity and hence the reproducibility of findings from animal research. We here aimed at proposing a new experimental strategy that transfers this logic into a single-laboratory setting. We systematically introduced heterogeneity into our study population by splitting an experiment into several ‘mini-experiments’ spread over different time points a few weeks apart. We hypothesised to observe improved reproducibility in such a ‘mini-experiment’ design in comparison to a conventionally standardised design, according to which all animals are tested at one specific point in time. By comparing both designs across independent replicates, we could indeed show that the use of such a ‘mini-experiment’ design improved the reproducibility and accurate detection of exemplary treatment effects (behavioural and physiological differences between four mouse strains) in about half of all investigated strain comparisons. Thus, we successfully implemented and empirically validated an easy-to-handle strategy to tackle poor reproducibility in single-laboratory studies. Since other experiments within different life science disciplines share the main characteristics with the investigation reported here, these studies are likely to also benefit from this approach.

A reaction norm perspective on reproducibility

“…Indeed, over the last decades several authors have emphasized and diligently advocated the use of mixed effect models for multi-centre studies (Localio et al. 2001 ; Kahan and Morris 2013 ) and meta analyses (Freeman et al. 1986 ).…”

Data variability in standardised cell culture experiments

“…Our analysis of NCI60 data, which is of unprecedented depth and provided by a world-leading institution with unprecedented transparency, indicates that data variation remains very high even under ideal conditions that the vast majority of research groups will not be able to afford. Hence, our data suggest that experiment heterogenization, the testing of a hypothesis in many different (experimental) systems and datasets and different laboratories [72–79], is a much better strategy to generate robust and meaningful data.…”

Comparison of single and repeated dosing of anti-inflammatory human umbilical cord mesenchymal stromal cells in a mouse model of polymicrobial sepsis

“…It should be acknowledged that data from animals that failed to survive to 48 hours could not be acquired and it is possible that analyses at one or more earlier time-points could provide a more distinctive responder/nonresponder separation. Nonetheless, this analysis highlights the complexity of inter-individual variation that is inherent to animal models of sepsis and cell therapies even with close attention to principles of good experimental design (44,45) and that re ects similar challenges faced in the clinical application of novel therapies to sepsis (18,20,38,46,47).…”