Test data reuse for evaluation of adaptive machine learning algorithms: over-fitting to a fixed 'test' dataset and a potential solution

Gossmann, Alexej; Pezeshk, Aria; Sahiner, Berkman

doi:10.1117/12.2293818

Cited by 10 publications

(9 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…If the hypothesis is not powerful enough to describe the data, then there will be an issue of underfitting . On the other hand, if the selected hypothesis is over complicated, then the model will learn from not only the inherent trend of the data but also the noise, which end up with overfitting . The model should be carefully selected considering three factors: the complexity of the hypothesis, the complexity of the training data, and the generalization performance on new examples .…”

Section: Approaches In Computational Materials Sciencementioning

confidence: 99%

“…[24][25][26] On the other hand, if the selected hypothesis is over complicated, then the model will learn from not only the inherent trend of the data but also the noise, which end up with overfitting. [27][28][29] The model should be carefully selected considering three factors: the complexity of the hypothesis, the complexity of the training data, and the generalization performance on new examples. [30][31][32] A set of assumptions that work well in one domain may work poorly in another.…”

Section: Approaches In Computational Materials Sciencementioning

confidence: 99%

See 1 more Smart Citation

Machine learning and artificial neural network accelerated computational discoveries in materials science

Yang

Hou

Jiang

et al. 2019

WIREs Comput Mol Sci

View full text Add to dashboard Cite

Artificial intelligence (AI) has been referred to as the "fourth paradigm of science," and as part of a coherent toolbox of data-driven approaches, machine learning (ML) dramatically accelerates the computational discoveries. As the machinery for ML algorithms matures, significant advances have been made not only by the mainstream AI researchers, but also those work in computational materials science. The number of ML and artificial neural network (ANN) applications in the computational materials science is growing at an astounding rate. This perspective briefly reviews the state-of-the-art progress in some supervised and unsupervised methods with their respective applications. The characteristics of primary ML and ANN algorithms are first described. Then, the most critical applications of AI in computational materials science such as empirical interatomic potential development, ML-based potential, property predictions, and molecular discoveries using generative adversarial networks (GAN) are comprehensively reviewed. The central ideas underlying these ML applications are discussed, and future directions for integrating ML with computational materials science are given. Finally, a discussion on the applicability and limitations of current ML techniques and the remaining challenges are summarized.

show abstract

Section: Approaches In Computational Materials Sciencementioning

confidence: 99%

Section: Approaches In Computational Materials Sciencementioning

confidence: 99%

Machine learning and artificial neural network accelerated computational discoveries in materials science

Yang

Hou

Jiang

et al. 2019

WIREs Comput Mol Sci

View full text Add to dashboard Cite

show abstract

“…Lockbox data site D lb : Lockbox [13] data site refers to data sites which the analyst from the openbox side can not access by any means. In practice, lockbox correspond to data sites that could not contribute in the process of building a machine learning model due to various reasons, but are likely to participate in the future or simply benefit from the model built.…”

Section: Terminology and Notationmentioning

confidence: 99%

“…Thresholdout Family: [7] showes that differential privacy is deeply associated with model generalization and propose the Thresholdout algorithm to avoid overfitting on the validation set due to repetitive usage. [13] extends the instance wise Thresholdout to AUC measures. However, these methods rely on the i.i.d assumption of data which does not fit our scenario here.…”

Section: Related Workmentioning

confidence: 99%

High Dimensional Restrictive Federated Model Selection with Multi-objective Bayesian Optimization over Shifted Distributions

Sun

Bommert

Pfisterer

et al. 2019

Advances in Intelligent Systems and Computing

View full text Add to dashboard Cite

A novel machine learning optimization process coined Restrictive Federated Model Selection (RFMS) is proposed under the scenario, for example, when data from healthcare units can not leave the site it is situated on and it is forbidden to carry out training algorithms on remote data sites due to either technical or privacy and trust concerns. To carry out a clinical research in this scenario, an analyst could train a machine learning model only on local data site, but it is still possible to execute a statistical query at a certain cost in the form of sending a machine learning model to some of the remote data sites and get the performance measures as feedback, maybe due to prediction being usually much cheaper. Compared to federated learning, which is optimizing the model parameters directly by carrying out training across all data sites, RFMS trains model parameters only on one local data site but optimizes hyper parameters across other data sites jointly since hyper-parameters play an important role in machine learning performance. The aim is to get a Pareto optimal model with respective to both local and remote unseen prediction losses, which could generalize well across data sites. In this work, we specifically consider high dimensional data with different distributions over data sites. As an initial investigation, Bayesian Optimization especially multi-objective Bayesian Optimization is used to guide an adaptive hyper-parameter optimization process to select models under the RFMS scenario. Empirical results shows that solely using the local data site to tune hyper-parameters generalizes poorly across data sites, compared to methods that utilize the local and remote performances. Furthermore, in terms of hypervolumes, multi-objective Bayesian Optimization algorithms show increased performance across multiple data sites among other candiates. Local:D obRemote: Dcu... ...

show abstract

“…Acquiring enough independent test data to validate each model, although the ideal, can become infeasible. An important and ongoing research area with many open questions is whether and how some test data can best be reused for validating future models (Dwork et al ., 2015; Gossmann et al ., 2018; Lee and Lee, 2020; Roelofs et al ., 2019; U.S. Food and Drug Administration, 2020a).…”

mentioning

confidence: 99%

Discussion on “Approval policies for modifications to machine learning‐based software as a medical device: A study of bio‐creep” by Jean Feng, Scott Emerson, and Noah Simon

et al. 2020

Self Cite

View full text Add to dashboard Cite

Test data reuse for evaluation of adaptive machine learning algorithms: over-fitting to a fixed 'test' dataset and a potential solution

Cited by 10 publications

References 20 publications

Machine learning and artificial neural network accelerated computational discoveries in materials science

Machine learning and artificial neural network accelerated computational discoveries in materials science

High Dimensional Restrictive Federated Model Selection with Multi-objective Bayesian Optimization over Shifted Distributions

Discussion on “Approval policies for modifications to machine learning‐based software as a medical device: A study of bio‐creep” by Jean Feng, Scott Emerson, and Noah Simon

Contact Info

Product

Resources

About