Issues in the Reproducibility of Deep Learning Results

Jean-Paul, Sartre; Elseify, Tarek; Obeid, Iyad

doi:10.1109/spmb47826.2019.9037840

Cited by 6 publications

(4 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Using such platform has the benefit of improving reusability and reproducibility of previous ML research. In another work, authors set the same random seed in order to measure difference in performance between different random seeds, and between PyTorch and TensorFlow [17]. They show a 7% difference between the random seeds; which is similar to the aforementioned results.…”

Section: Related Worksupporting

confidence: 73%

See 1 more Smart Citation

The challenge of reproducible ML: an empirical study on the impact of bugs

Rivera-Landos¹,

Khomh²,

Nikanjam³

2021

Preprint

View full text Add to dashboard Cite

Reproducibility is a crucial requirement in scientific research. When results of research studies and scientific papers have been found difficult or impossible to reproduce, we face a challenge which is called reproducibility crisis. Although the demand for reproducibility in Machine Learning (ML) is acknowledged in the literature, a main barrier is inherent nondeterminism in ML training and inference. In this paper, we establish the fundamental factors that cause non-determinism in ML systems. A framework, ReproduceML, is then introduced for deterministic evaluation of ML experiments in a real, controlled environment. ReproduceML allows researchers to investigate software configuration effects on ML training and inference. Using ReproduceML, we run a case study: investigation of the impact of bugs inside ML libraries on performance of ML experiments. This study attempts to quantify the impact that the occurrence of bugs in a popular ML framework, PyTorch, has on the performance of trained models. To do so, a comprehensive methodology is proposed to collect buggy versions of ML libraries and run deterministic ML experiments using ReproduceML. Our initial finding is that there is no evidence based on our limited dataset to show that bugs which occurred in PyTorch do affect the performance of trained models. The proposed methodology as well as ReproduceML can be employed for further research on non-determinism and bugs.

show abstract

Section: Related Worksupporting

confidence: 73%

“…A similar study for ML applied in medicine shows a 9% reproducibility [15]. Other researchers have also called for better reproducibility in ML [2], [16] and DL [17], [18].…”

Section: Introductionmentioning

confidence: 84%

The challenge of reproducible ML: an empirical study on the impact of bugs

Rivera-Landos¹,

Khomh²,

Nikanjam³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…It would be worth conducting further research to determine the extent to which the lack of code and data sharing is a general trend, rather than specific to the domain of CVD. It is understandable that patient data is sensitive and the developed algorithms are intellectual properties [19]. However, sharing does not necessarily mean providing unlimited access for free, rather using a set of protocols and appropriate licenses that enable other researchers to use and cite the work as needed.…”

Section: Discussionmentioning

confidence: 99%

Assessing the FAIRness of Deep Learning Models in Cardiovascular Disease Using Computed Tomography Images: Data and Code Perspective

Shiferaw

Zeleke

Waltemath

2023

Caring Is Sharing – Exploiting the Value in Data for Health and Innovation

View full text Add to dashboard Cite

The interest in the application of AI in medicine has intensely increased over the past decade with most of the changes in the past five years. Most recently, the application of deep learning algorithms in prediction and classification of cardiovascular diseases (CVD) using computed tomography (CT) images showed promising results. The notable and exciting advancement in this area of study is, however, associated with different challenges related to the findability (F), accessibility(A), interoperability(I), reusability(R) of both data and source code. The aim of this work is to identify reoccurring missing FAIR-related features and to assess the level of FAIRness of data and models used to predict/diagnose cardiovascular diseases from CT images. We evaluated the FAIRness of data and models in published studies using the RDA (Research Data Alliance) FAIR Data maturity model and FAIRshake toolkit. The finding showed that although AI is anticipated to bring ground breaking solutions for complex medical problems, the findability, accessibility, interoperability and reusability of data/metadata/code is still a prominent challenge.

show abstract

“…At minimum, the specifics about the CPU and GPU. This is for indicating the amount of compute necessary for the project, but also for the sake of replicability issues due to the non-deterministic nature of the GPU (Jean-Paul et al, 2019;. Moreover, Dodge et al (2019) demonstrate that test performance scores alone are insufficient for claiming the dominance of a model over another, and argue for reporting additional performance details on validation data as a function of computation budget, which can also estimate the amount of computation required to obtain a given accuracy.…”

Section: Hardware Requirementsmentioning

confidence: 99%

Experimental Standards for Deep Learning in Natural Language Processing Research

Ulmer¹,

Bassignana²,

Müller-Eberstein³

et al. 2022

Preprint

View full text Add to dashboard Cite

The field of Deep Learning (DL) has undergone explosive growth during the last decade, with a substantial impact on Natural Language Processing (NLP) as well. Yet, as with other fields employing DL techniques, there has been a lack of common experimental standards compared to more established disciplines. Starting from fundamental scientific principles, we distill ongoing discussions on experimental standards in DL into a single, widely-applicable methodology. Following these best practices is crucial to strengthening experimental evidence, improve reproducibility and enable scientific progress. These standards are further collected in a public repository to help them transparently adapt to future needs.

show abstract

Issues in the Reproducibility of Deep Learning Results

Cited by 6 publications

References 6 publications

The challenge of reproducible ML: an empirical study on the impact of bugs

The challenge of reproducible ML: an empirical study on the impact of bugs

Assessing the FAIRness of Deep Learning Models in Cardiovascular Disease Using Computed Tomography Images: Data and Code Perspective

Experimental Standards for Deep Learning in Natural Language Processing Research

Contact Info

Product

Resources

About