The Prevalence of Errors in Machine Learning Experiments

Abstract

Context: Conducting experiments is central to research machine learning research to benchmark, evaluate and compare learning algorithms. Consequently it is important we conduct reliable, trustworthy experiments. Objective: We investigate the incidence of errors in a sample of machine learning experiments in the domain of software defect prediction. Our focus is simple arithmetical and statistical errors. Method : We analyse 49 papers describing 2456 individual experimental results from a previously undertaken systematic review comparing supervised and unsupervised defect prediction classifiers. We extract the confusion matrices and test for relevant constraints, e.g., the marginal probabilities must sum to one. We also check for multiple statistical significance testing errors. Results: We find that a total of 22 out of 49 papers contain demonstrable errors. Of these 7 were statistical and 16 related to confusion matrix inconsistency (one paper contained both classes of error). Conclusions: Whilst some errors may be of a relatively trivial nature, e.g., transcription errors their presence does not engender confidence. We strongly urge researchers to follow open science principles so errors can be more easily be detected and corrected, thus as a community reduce this worryingly high error rate with our computational experiments.

Remaining Useful Life Estimation of Bearings

“…Aggregate the results to provide a summary of the performance so that one can determine if the method has, on average, better performance for a specific single-mode or multi-mode problem. Use formal methods to compare the result (Shepperd et al, 2019) (for example the Bonferroni correction, Benjamini-Hochberg false discovery rate estimate and Nemenyi post hoc procedure).…”

“…To avoid leakage, test data set instances must not, under any circumstance, be used for training (see sections 8.3 and 6.2). If many models are generated during hyper-parameter tweaking, then an appropriate method must be used to select the best model (Shepperd et al, 2019) (section 7.3), or alternately, report on the robustness of the models (using for example sensitivity analysis (Cortez & Embrechts, 2013) or at the very least simple aggregates such as median, inter-quartile range, minimum and maximum (Giles & Lawrence, 1997). This avoids a form of data snooping where the selected model has a particularly high performance that is due to chance alone (6.3).…”

“…Researchers have concluded that many experimental results are unreliable (Shepperd et al, 2019;Ioannidis, 2005). Errors have been found, some of which may be attributed to simple transcription errors.…”

“…Errors have been found, some of which may be attributed to simple transcription errors. This is in part due to the complex and chaotic process that includes data pre-processing, feature generation and selection, hyper-parameter tuning via crossvalidation, metric selection and model performance comparison (Shepperd et al, 2019).…”

“…Several researchers concerned with this issue have done meta-International Journal of Prognostics and Health Management, ISSN2153-2648, 2020 010 analysis on related work to try and identify what errors are committed and how one may go about avoiding these. This includes studying the validity of null hypothesis significance testing (Colquhoun, 2018), checking simple integrity constraints (arithmetical and statistical errors) (Shepperd et al, 2019) and examining the correctness of the experimental design and analysis (Ioannidis, 2005). In this work we will look at the machine learning experimental design in RUL estimation with an emphasis on the comparative analysis of the generated prediction models.…”