An Approach for Validating Quality of Datasets for Machine Learning

Ding, Junhua; Li, Xinchuan

doi:10.1109/bigdata.2018.8622640

Cited by 17 publications

(10 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This is a truism in ML; including local data into a general model invariably improves local performance [25]. Less obvious is how much local data is needed [26], and what quality [diversity, accuracy] of the local data is needed [27]. Another question is how best to utilise local data: via incorporation into an existing model (transfer learning [28]), or by building a new model using both the general data (here the MARCO training set) and local data [29].…”

Section: Plos Onementioning

confidence: 99%

Moving beyond MARCO

Rosa

Watkins²,

Newman³

2023

PLoS ONE

View full text Add to dashboard Cite

The use of imaging systems in protein crystallisation means that the experimental setups no longer require manual inspection to determine the outcome of the trials. However, it leads to the problem of how best to find images which contain useful information about the crystallisation experiments. The adoption of a deeplearning approach in 2018 enabled a four-class machine classification system of the images to exceed human accuracy for the first time. Underpinning this was the creation of a labelled training set which came from a consortium of several different laboratories. The MARCO classification model does not have the same accuracy on local data as it does on images from the original test set; this can be somewhat mitigated by retraining the ML model and including local images. We have characterized the image data used in the original MARCO model, and performed extensive experiments to identify training settings most likely to enhance the local performance of a MARCO-dataset based ML classification model.

show abstract

Section: Plos Onementioning

confidence: 99%

Moving beyond MARCO

Rosa

Watkins²,

Newman³

2023

PLoS ONE

View full text Add to dashboard Cite

show abstract

“…Another technique is metamorphic testing, which deals with generic transformations of the relationship between inputs and outputs. This makes it possible to validate the model's diversity, fidelity, and veracity in relation to the dataset [28].…”

Section: Related Workmentioning

confidence: 99%

Dataset Quality Assessment with Permutation Testing Showcased on Network Traffic Datasets

Wasielewska¹,

Soukup²,

Čejka³

et al. 2022

Preprint

View full text Add to dashboard Cite

<p>Intelligent and autonomous networks require precise and fast mechanisms that ensure error-free and efficient operation. Modern solutions are increasingly based on artificial intelligence, in particular on machine learning, to reliably process huge amounts of data. Therefore, high-quality datasets are essential to train machine learning models. Unfortunately, the problem of assessing the quality of datasets is very challenging and often overlooked. This paper proposes a method for assessing the dataset quality in the context of binary classification. It is based on permutation testing and examines the strength of the relationship between observations and labels. Experiments carried out on simulated and real network datasets show that the method is sensitive to detect errors/mislabels in the labeled dataset. We also present theoretical considerations justifying our results.</p>

show abstract

“…7 Max Planck Institute for Molecular Genetics, Berlin, Germany. 8 Alacris Theranostics GmbH, Berlin, Germany. 9 School of Medicine Biostatistics and Medical Informatics Dept., Acibadem University, Istanbul, Turkey.…”

Section: Fundingmentioning

confidence: 99%

“…For example, the first AI-based device to receive market authorization from the FDA was assessed with a large prospective comparative clinical trial including 900 patients from multiple sites [4]. AI technologies must satisfy stringent regulations for approval as medical devices, because (1) the decision support provided is optimized and personalized continuously in real time, according to the phenotype of the patient [7]; (2) the performance of AI depends strongly on the training datasets used [8], resulting in a large risk of AI performing less well in real practice [9][10][11] or on another group of patients or institutions [9]. It is, therefore, essential to assess the performance and safety of AI before its introduction into routine clinical use.…”

Section: Introductionmentioning

confidence: 99%

A framework for validating AI in precision medicine: considerations from the European ITFoC consortium

Tsopra

Luchinat

Alberghina

et al. 2021

BMC Med Inform Decis Mak

View full text Add to dashboard Cite

Background Artificial intelligence (AI) has the potential to transform our healthcare systems significantly. New AI technologies based on machine learning approaches should play a key role in clinical decision-making in the future. However, their implementation in health care settings remains limited, mostly due to a lack of robust validation procedures. There is a need to develop reliable assessment frameworks for the clinical validation of AI. We present here an approach for assessing AI for predicting treatment response in triple-negative breast cancer (TNBC), using real-world data and molecular -omics data from clinical data warehouses and biobanks. Methods The European “ITFoC (Information Technology for the Future Of Cancer)” consortium designed a framework for the clinical validation of AI technologies for predicting treatment response in oncology. Results This framework is based on seven key steps specifying: (1) the intended use of AI, (2) the target population, (3) the timing of AI evaluation, (4) the datasets used for evaluation, (5) the procedures used for ensuring data safety (including data quality, privacy and security), (6) the metrics used for measuring performance, and (7) the procedures used to ensure that the AI is explainable. This framework forms the basis of a validation platform that we are building for the “ITFoC Challenge”. This community-wide competition will make it possible to assess and compare AI algorithms for predicting the response to TNBC treatments with external real-world datasets. Conclusions The predictive performance and safety of AI technologies must be assessed in a robust, unbiased and transparent manner before their implementation in healthcare settings. We believe that the consideration of the ITFoC consortium will contribute to the safe transfer and implementation of AI in clinical settings, in the context of precision oncology and personalized care.

show abstract

An Approach for Validating Quality of Datasets for Machine Learning

Cited by 17 publications

References 18 publications

Moving beyond MARCO

Moving beyond MARCO

Dataset Quality Assessment with Permutation Testing Showcased on Network Traffic Datasets

A framework for validating AI in precision medicine: considerations from the European ITFoC consortium

Contact Info

Product

Resources

About