A comprehensive survey on regularization strategies in machine learning

Tian, Yingjie; Zhang, Yuqi

doi:10.1016/j.inffus.2021.11.005

Cited by 141 publications

(58 citation statements)

References 83 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There are numerous strategies to tackle this issue, such as early stopping (stop the training before the model incorporates noise); expand the training set (more data implies a more accurate model); feature selection (this identifies the most relevant features to be learned, ignoring redundant ones); ensemble methods (that aggregate the output of a set of classifiers, selecting the best output by a voting process); and, finally, regularisation (that, in general, limits the amount of variance in the model by penalising input parameters with large coefficients). Regularisation methods have received great attention from recent studies [ 61 ], since they are solely related to the algorithms, and not to data quality or classifier competitions. One particular regularisation method worth mentioning in this review is Dropout , that ignores randomly chosen neurons (with a certain probability) during the training phase, so that a reduced network is obtained as a result.…”

Section: Background Knowledgementioning

confidence: 99%

A Survey of Underwater Acoustic Data Classification Methods Using Deep Learning for Shoreline Surveillance

Domingos

Santos

Skelton

et al. 2022

Sensors

View full text Add to dashboard Cite

This paper presents a comprehensive overview of current deep-learning methods for automatic object classification of underwater sonar data for shoreline surveillance, concentrating mostly on the classification of vessels from passive sonar data and the identification of objects of interest from active sonar (such as minelike objects, human figures or debris of wrecked ships). Not only is the contribution of this work to provide a systematic description of the state of the art of this field, but also to identify five main ingredients in its current development: the application of deep-learning methods using convolutional layers alone; deep-learning methods that apply biologically inspired feature-extraction filters as a preprocessing step; classification of data from frequency and time–frequency analysis; methods using machine learning to extract features from original signals; and transfer learning methods. This paper also describes some of the most important datasets cited in the literature and discusses data-augmentation techniques. The latter are used for coping with the scarcity of annotated sonar datasets from real maritime missions.

show abstract

Section: Background Knowledgementioning

confidence: 99%

A Survey of Underwater Acoustic Data Classification Methods Using Deep Learning for Shoreline Surveillance

Domingos

Santos

Skelton

et al. 2022

Sensors

View full text Add to dashboard Cite

show abstract

“…Batch size defines the number of inputs that will be propagated through the network each time. Batch normalization, as one of the common regularization strategies, aims to deal with noise data, the limited size of the training data, and the complexity of classifiers to avoid overfitting [49]. Using a smaller batch size requires less memory and results in faster training; however, setting the batch size too small will result in less accuracy for the estimate of the gradient.…”

Section: Hyperparameter Tuningmentioning

confidence: 99%

Multiclass anomaly detection in imbalanced structural health monitoring data using convolutional neural network

Zhao

Sadhu

Capretz

2022

J Infrastruct Preserv Resil

View full text Add to dashboard Cite

Structural health monitoring (SHM) system aims to monitor the in-service condition of civil infrastructures, incorporate proactive maintenance, and avoid potential safety risks. An SHM system involves the collection of large amounts of data and data transmission. However, due to the normal aging of sensors, exposure to outdoor weather conditions, accidental incidences, and various operational factors, sensors installed on civil infrastructures can get malfunctioned. A malfunctioned sensor induces significant multiclass anomalies in measured SHM data, requiring robust anomaly detection techniques as an essential data cleaning process. Moreover, civil infrastructure often has imbalanced anomaly data where most of the SHM data remain biased to a certain type of anomalies. This imbalanced time-series data causes significant challenges to the existing anomaly detection methods. Without proper data cleaning processes, the SHM technology does not provide useful insights even if advanced damage diagnostic techniques are applied. This paper proposes a hyperparameter-tuned convolutional neural network (CNN) for multiclass imbalanced anomaly detection (CNN-MIAD) modelling. The hyperparameters of the proposed model are tuned through a random search algorithm to optimize the performance. The effect of balancing the database is considered by augmenting the dataset. The proposed CNN-MIAD model is demonstrated with a multiclass time-series of anomaly data obtained from a real-life cable-stayed bridge under various cases of data imbalances. The study concludes that balancing the database with a time shift window to increase the database has generated the optimum results, with an overall accuracy of 97.74%.

show abstract

“…The (2, 1)-norm favors a small number of nonzero rows in the matrix W , therefore ensuring that the common features (most effective centers) will be selected. It should be noted that, Regularization techniques [10,11] proved to improve the generalization ability and therefore the performance of a model. A comprehensive study and a state-of-the-art review of the regularization strategies in machine learning is given in [10].…”

Section: Introductionmentioning

confidence: 99%

“…It should be noted that, Regularization techniques [10,11] proved to improve the generalization ability and therefore the performance of a model. A comprehensive study and a state-of-the-art review of the regularization strategies in machine learning is given in [10]. It is being used in different classification problems such as, image recognition [12], Underwater Acoustic Data Classification [11] e.t.c.…”

Section: Introductionmentioning

confidence: 99%

Supervised Regularized Multidimensional Scaling Using Weighted Stress Measure

Islam¹,

Bhuiyan²,

Jahan³

2022

ujam

View full text Add to dashboard Cite

Supervised Regularized Multidimensional Scaling (SRMDS), a non-linear variant of classical Multi-Dimensional Scaling (cMDS) is proposed recently which involves Radial basis function. The method is focused on the effective selection of centers of the radial basis functions in transforming data from a higher dimensional space to a lower dimension. The transformation matrix is determined by minimizing stress. Weights of components of the stress that are of great importance for classification of data got less focus in Supervised Regularized Multidimensional Scaling. In this article, we have investigated several forms of non-linear functions which may be used as weights of the stress measure. A new form of Z-shape weight function dependent on intraclass information of the dataset is introduced which prefers to preserve global structure of the dataset. The efficiency of the proposed approach is illustrated on several benchmarking datasets which shows that the weighted Supervised Regularized Multidimensional Scaling (WSRMDS) outperforms the base method and some other state of the art approaches such as Linear Discriminant Analysis (LDA), and Supervised Principal Component Analysis (SPCA). Observing the finding of this research, among different weight functions, Z-shape weight function is recommended to use since it works better then any other weight functions for most of the data sets.

show abstract

A comprehensive survey on regularization strategies in machine learning

Cited by 141 publications

References 83 publications

A Survey of Underwater Acoustic Data Classification Methods Using Deep Learning for Shoreline Surveillance

A Survey of Underwater Acoustic Data Classification Methods Using Deep Learning for Shoreline Surveillance

Multiclass anomaly detection in imbalanced structural health monitoring data using convolutional neural network

Supervised Regularized Multidimensional Scaling Using Weighted Stress Measure

Contact Info

Product

Resources

About