A Plea for Neutral Comparison Studies in Computational Sciences

Boulesteix, Anne‐Laure; Lauer, Sabine; Eugster, Manuel J. A.

doi:10.1371/journal.pone.0061562

Cited by 118 publications

(160 citation statements)

References 11 publications

Supporting

Mentioning

154

Contrasting

Order By: Relevance

“…In this setting, the comparison of existing classification methods using real microarray datasets has been the topic of a number of papers [11–18], which do not aim at demonstrating the superiority of a new method proposed by the authors: they consider only existing methods, thus implying a certain level of neutrality [19]. Most of these papers consider measures such as the error rate or the area under the curve to assess the performances of the classification methods.…”

Section: Motivating Examplementioning

confidence: 99%

Towards evidence-based computational statistics: lessons from clinical research on the role and design of real-data benchmark studies

Boulesteix

Wilson

Hapfelmeier³

2017

BMC Med Res Methodol

Self Cite

View full text Add to dashboard Cite

BackgroundThe goal of medical research is to develop interventions that are in some sense superior, with respect to patient outcome, to interventions currently in use. Similarly, the goal of research in methodological computational statistics is to develop data analysis tools that are themselves superior to the existing tools. The methodology of the evaluation of medical interventions continues to be discussed extensively in the literature and it is now well accepted that medicine should be at least partly “evidence-based”. Although we statisticians are convinced of the importance of unbiased, well-thought-out study designs and evidence-based approaches in the context of clinical research, we tend to ignore these principles when designing our own studies for evaluating statistical methods in the context of our methodological research.Main messageIn this paper, we draw an analogy between clinical trials and real-data-based benchmarking experiments in methodological statistical science, with datasets playing the role of patients and methods playing the role of medical interventions. Through this analogy, we suggest directions for improvement in the design and interpretation of studies which use real data to evaluate statistical methods, in particular with respect to dataset inclusion criteria and the reduction of various forms of bias. More generally, we discuss the concept of “evidence-based” statistical research, its limitations and its impact on the design and interpretation of real-data-based benchmark experiments.ConclusionWe suggest that benchmark studies—a method of assessment of statistical methods using real-world datasets—might benefit from adopting (some) concepts from evidence-based medicine towards the goal of more evidence-based statistical research.

show abstract

Section: Motivating Examplementioning

confidence: 99%

Towards evidence-based computational statistics: lessons from clinical research on the role and design of real-data benchmark studies

Boulesteix

Wilson

Hapfelmeier³

2017

BMC Med Res Methodol

Self Cite

View full text Add to dashboard Cite

show abstract

“…We selected comparison studies (i) whose aim was not to establish the superiority of a "new method", hence warranting a certain level of neutrality Boulesteix and Eugster, 2012), (ii) …”

Section: Considered Studiesmentioning

confidence: 99%

A Statistical Framework for Hypothesis Testing in Real Data Comparison Studies

Boulesteix¹,

Hable²,

Lauer³

et al. 2015

The American Statistician

Self Cite

View full text Add to dashboard Cite

In computational sciences, including computational statistics, machine learning, and bioinformatics, most abstracts of articles presenting new supervised learning methods end with a sentence like "our method performed better than existing methods on real data sets", e.g. in terms of error rate. However, these claims are often not based on proper statistical tests and, if such tests are performed (as usual in the machine learning literature), the tested hypothesis is not clearly defined and poor attention is devoted to the type I and type II error. In the present paper we aim to fill this gap by providing a proper statistical framework for hypothesis tests comparing the performance of supervised learning methods based on several real data sets with unknown underlying distribution. After giving a statistical interpretation of ad-hoc tests commonly performed by machine learning scientists, we devote special attention to power issues and suggest a simple method to determine the number of data sets to be included in a comparison study to reach an adequate power. These methods are illustrated through three comparison studies from the literature and an exemplary benchmarking study using gene expression microarray data. All our results can be reproduced using R-codes and data sets available from the companion website

show abstract

“…Comparison studies published in literature often include a large number of methods but a relatively small number of datasets [5], yielding an ill-posed problem as far as statistical interpretation of benchmarking results are concerned. In the present paper we take an opposite approach: we focus on only two methods for the reasons outlined above but design our benchmarking experiments in such a way that it yields solid evidence.…”

Section: Introductionmentioning

confidence: 99%

“…Neutrality and equal expertise would be much more difficult if not impossible to ensure if several variants of RF (including tuning strategies) and logistic regression were included in the study. Further discussions of the concept of authors’ neutrality can be found elsewhere [5, 11]. …”

Section: Introductionmentioning

confidence: 99%

Random forest versus logistic regression: a large-scale benchmark experiment

2018

Self Cite

View full text Add to dashboard Cite

Background and goalThe Random Forest (RF) algorithm for regression and classification has considerably gained popularity since its introduction in 2001. Meanwhile, it has grown to a standard classification approach competing with logistic regression in many innovation-friendly scientific fields.ResultsIn this context, we present a large scale benchmarking experiment based on 243 real datasets comparing the prediction performance of the original version of RF with default parameters and LR as binary classification tools. Most importantly, the design of our benchmark experiment is inspired from clinical trial methodology, thus avoiding common pitfalls and major sources of biases.ConclusionRF performed better than LR according to the considered accuracy measured in approximately 69% of the datasets. The mean difference between RF and LR was 0.029 (95%-CI =[0.022,0.038]) for the accuracy, 0.041 (95%-CI =[0.031,0.053]) for the Area Under the Curve, and − 0.027 (95%-CI =[−0.034,−0.021]) for the Brier score, all measures thus suggesting a significantly better performance of RF. As a side-result of our benchmarking experiment, we observed that the results were noticeably dependent on the inclusion criteria used to select the example datasets, thus emphasizing the importance of clear statements regarding this dataset selection process. We also stress that neutral studies similar to ours, based on a high number of datasets and carefully designed, will be necessary in the future to evaluate further variants, implementations or parameters of random forests which may yield improved accuracy compared to the original version with default values.Electronic supplementary materialThe online version of this article (10.1186/s12859-018-2264-5) contains supplementary material, which is available to authorized users.

show abstract

A Plea for Neutral Comparison Studies in Computational Sciences

Cited by 118 publications

References 11 publications

Towards evidence-based computational statistics: lessons from clinical research on the role and design of real-data benchmark studies

Towards evidence-based computational statistics: lessons from clinical research on the role and design of real-data benchmark studies

A Statistical Framework for Hypothesis Testing in Real Data Comparison Studies

Random forest versus logistic regression: a large-scale benchmark experiment

Contact Info

Product

Resources

About