A Replication Crisis in Methodological Research?

Boulesteix, Anne‐Laure; Hoffmann, Sabine; Charlton, Alethea; Seibold, Heidi

doi:10.1111/1740-9713.01444

Cited by 34 publications

(34 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As such, the development of new computational methods is an active field of research, a situation which constantly introduces new methods into the literature, often in a comparison with existing methods. Yet despite the frequency of these types of papers, there is a surprising lack of guidance on the appropriate design and reporting of studies presenting and evaluating new computational methods [ 1 , 2 ]. It is not clear how new methods and their performances should be described and how studies comparing performances of methods should be designed.…”

Section: Introductionmentioning

confidence: 99%

On the optimistic performance evaluation of newly introduced bioinformatic methods

Buchka

Hapfelmeier²,

Geeleher

et al. 2021

Genome Biol

View full text Add to dashboard Cite

Most research articles presenting new data analysis methods claim that “the new method performs better than existing methods,” but the veracity of such statements is questionable. Our manuscript discusses and illustrates consequences of the optimistic bias occurring during the evaluation of novel data analysis methods, that is, all biases resulting from, for example, selection of datasets or competing methods, better ability to fix bugs in a preferred method, and selective reporting of method variants. We quantitatively investigate this bias using an example from epigenetic analysis: normalization methods for data generated by the Illumina HumanMethylation450K BeadChip microarray.

show abstract

Section: Introductionmentioning

confidence: 99%

On the optimistic performance evaluation of newly introduced bioinformatic methods

Buchka

Hapfelmeier²,

Geeleher

et al. 2021

Genome Biol

View full text Add to dashboard Cite

show abstract

“…The characterisation of the different types of errors and their frequencies that we report might be of use to designing better simulation studies that test error detection routines. Such simulation studies should mimic the types of errors that occur in datasets, otherwise results may have little generalisability when applied to datasets that have different error structures (49). Merely simulating normally distributed errors without age (date) errors, keystroke errors (29), duplications, internally inconsistent values etc is unrealistic, and such studies are unlikely to be a useful test of the performance of the method.…”

Section: Discussionmentioning

confidence: 99%

Screening & diagnosing errors in longitudinal measures of body size

Wills

2020

Preprint

View full text Add to dashboard Cite

This paper presents a novel multi-step automated algorithm to screen for errors in longitudinal height and weight data and describes the frequency and characteristics of errors in three datasets. It also offers a taxonomy of published cleaning routines from a scoping review.Illustrative data are from three Norwegian retrospective cohorts containing 87,792 assessments (birth to 14y) from 8,428 children. Each has different data pipelines, quality control and data structure. The algorithm contains 43 steps split into 3 sections; (a) dates, (b) Identifiable data entry errors, (c) biologically impossible/ implausible change, and uses logic checks, and cross-sectional and longitudinal routines. The WHO cross-sectional approach was also applied as a comparison.Published cleaning routines were taxonomized by their design, the marker used to screen errors, the reference threshold and how threshold was selected. Fully automated error detection was not possible without false positives or reduced sensitivity. Error frequencies in the cohorts were 0.4%, 2.1% and 2.4% of all assessments, and the percentage of children with ≥1 error was 4.1%, 13.4% and 15.3%. In two of the datasets, >2/3s of errors could be classified as inliers (within ±3SD scores). Children with errors had a similar distribution of HT and WT to those without error. The WHO cross-sectional approach lacked sensitivity (range 0-55%), flagged many false positives (range: 7-100%) and biased estimates of overweight and thinness.Elements of this algorithm may have utility for built-in data entry rules, data harmonisation and sensitivity analyses. The reported error frequencies and structure may also help design more realistic simulation studies to test routines. Multi-step distribution-wide algorithmic approaches are recommended to systematically screen and document the wide range of ways in which errors can occur and to maximise sensitivity for detecting errors, naive cross-sectional trimming as a stand-alone method may do more harm than good.

show abstract

“…The very people who often provide statistical advice to applied researchers and lament the brevity of method sections or lack of rigorous reporting fall prey to the same trappings. Wouldn't it be nice if simulation studies contained not only guidance for data analysis and study design but were additionally exemplary in transparency, data and code sharing (Boulesteix et al, 2020)?…”

Section: ) Research For Research Should Lead By Examplementioning

confidence: 99%

It's time! 10 + 1 reasons we should start replicating simulation studies

A¹,

Olo²,

Morris³

et al. 2021

Preprint

View full text Add to dashboard Cite

The quantitative analysis of research data is a core element of empirical research. The performance of statistical methods that are used for analyzing empirical data can be evaluated and compared with computer simulations. A single simulation study can determine the analyses of thousands of empirical studies to follow. With great power comes great responsibility. Here, we argue that this responsibility includes replication of simulation studies to ensure a sound foundation for data analytical decisions. Furthermore, being designed, run and reported by humans, simulation studies face challenges similar to other experimental empirical research and hence should not be exempt from replication attempts. We highlight that the potential replicability of simulation studies is an opportunity and a luxury that quantitative methodology as a field should proudly embrace and use as a chance to lead by example.

show abstract

A Replication Crisis in Methodological Research?

Cited by 34 publications

References 13 publications

On the optimistic performance evaluation of newly introduced bioinformatic methods

On the optimistic performance evaluation of newly introduced bioinformatic methods

Screening & diagnosing errors in longitudinal measures of body size

It's time! 10 + 1 reasons we should start replicating simulation studies

Contact Info

Product

Resources

About