A systematic review on overfitting control in shallow and deep neural networks

Bejani, Mohammad Mahdi; Ghatee, Mehdi

doi:10.1007/s10462-021-09975-1

Cited by 261 publications

(108 citation statements)

References 138 publications

Supporting

Mentioning

104

Contrasting

Unclassified

Order By: Relevance

“…We can also observe that GL obtains a greater classification accuracy improvement (19.2%) over the other methods on all datasets. Neural networks have a high capacity to overfit the untrustworthy small tabular data [18], [57], which causes poor generalization ability on test in-distribution samples. GL alleviates this problem by adaptively applying ground-truth labels and complementary labels for in-and out-of-distribution samples, respectively, where the complementary labels decrease the risk of providing incorrect label information.…”

Section: Comparison Resultsmentioning

confidence: 99%

Gray Learning from Non-IID Data with Out-of-distribution Samples

Zhao¹,

Cao²,

Wang³

2022

Preprint

View full text Add to dashboard Cite

The quality of the training data annotated by experts cannot be guaranteed, even more so for non-IID data consisting of both in-and out-of-distribution samples (i.e., in-distribution and out-of-distribution samples hold different distributions). Experts may mistakenly annotate out-of-distribution samples the same as in-distribution samples, incurring untrustworthy ground-truth labels. Learning such non-IID data mixing in-and out-of-distribution samples with untrustworthy labels significantly challenges both shallow and deep learning, with no relevant work reported. It would be possible to identify trustworthy complementary labels of a sample indicating which classes it does not belong to, because both in-and out-of-distribution samples do not belong to the classes except those corresponding to the ground-truth label. With this insight, we propose a novel gray learning approach to robustly learn from non-IID data with both in-and out-of-distribution samples. Due to the uncertain distributions of training samples, we reject the complementary labels for low-confidence inputs while mapping high-confidence inputs to the ground-truth labels in training. Building on the statistical learning theory, we derive the generalization error which shows that gray learning achieves a tight bound on the non-IID data. Extensive experiments show that our method provides significant improvement over alternative methods from robust statistics.

show abstract

Section: Comparison Resultsmentioning

confidence: 99%

Gray Learning from Non-IID Data with Out-of-distribution Samples

Zhao¹,

Cao²,

Wang³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The feedforward model architecture used was limited in the number of hidden layers and neurons per hidden layer to allow for the execution of a variety of different model evaluations and the time required to train one model per evaluation. Further, more complex models might have run the risk of overfitting due to a larger number of trainable hyperparameters [43].…”

Section: Feedforward Neural Networkmentioning

confidence: 99%

Evaluation of sEMG Signal Features and Segmentation Parameters for Limb Movement Prediction Using a Feedforward Neural Network

et al. 2022

View full text Add to dashboard Cite

Limb movement prediction based on surface electromyography (sEMG) for the control of wearable robots, such as active orthoses and exoskeletons, is a promising approach since it provides an intuitive control interface for the user. Further, sEMG signals contain early information about the onset and course of limb movements for feedback control. Recent studies have proposed machine learning-based modeling approaches for limb movement prediction using sEMG signals, which do not necessarily require domain knowledge of the underlying physiological system and its parameters. However, there is limited information on which features of the measured sEMG signals provide the best prediction accuracy of machine learning models trained with these data. In this work, the accuracy of elbow joint movement prediction based on sEMG data using a simple feedforward neural network after training with different single- and multi-feature sets and data segmentation parameters was compared. It was shown that certain combinations of time-domain and frequency-domain features, as well as segmentation parameters of sEMG data, improve the prediction accuracy of the neural network as compared to the use of a standard feature set from the literature.

show abstract

“…A model with too little capacity cannot learn the given problem, whereas a model with too much capacity can learn it too well and overfit the training dataset; thus, such a model does not have the ability to generalize the knowledge well [ 38 ]. Thus, the selection of a model size that maximizes generalization is an important topic that has been given a lot of attention in the last years as the field of cognitive neuroscience develops [ 39 ].…”

Section: Introductionmentioning

confidence: 99%

Coarse-Grained Neural Network Model of the Basal Ganglia to Simulate Reinforcement Learning Tasks

Drapała

Frydecka

2022

Brain Sciences

View full text Add to dashboard Cite

Computational models of the basal ganglia (BG) provide a mechanistic account of different phenomena observed during reinforcement learning tasks performed by healthy individuals, as well as by patients with various nervous or mental disorders. The aim of the present work was to develop a BG model that could represent a good compromise between simplicity and completeness. Based on more complex (fine-grained neural network, FGNN) models, we developed a new (coarse-grained neural network, CGNN) model by replacing layers of neurons with single nodes that represent the collective behavior of a given layer while preserving the fundamental anatomical structures of BG. We then compared the functionality of both the FGNN and CGNN models with respect to several reinforcement learning tasks that are based on BG circuitry, such as the Probabilistic Selection Task, Probabilistic Reversal Learning Task and Instructed Probabilistic Selection Task. We showed that CGNN still has a functionality that mirrors the behavior of the most often used reinforcement learning tasks in human studies. The simplification of the CGNN model reduces its flexibility but improves the readability of the signal flow in comparison to more detailed FGNN models and, thus, can help to a greater extent in the translation between clinical neuroscience and computational modeling.

show abstract

A systematic review on overfitting control in shallow and deep neural networks

Cited by 261 publications

References 138 publications

Gray Learning from Non-IID Data with Out-of-distribution Samples

Gray Learning from Non-IID Data with Out-of-distribution Samples

Evaluation of sEMG Signal Features and Segmentation Parameters for Limb Movement Prediction Using a Feedforward Neural Network

Coarse-Grained Neural Network Model of the Basal Ganglia to Simulate Reinforcement Learning Tasks

Contact Info

Product

Resources

About