Batch gradient training method with smoothing $$\boldsymbol{\ell}_{\bf 0}$$ ℓ 0 regularization for feedforward neural networks

Zhang, Huisheng; Tang, Yanli; Liu, Xiaodong

doi:10.1007/s00521-014-1730-x

Cited by 19 publications

(7 citation statements)

References 33 publications

(36 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In literature, there is no standard threshold value to remove unnecessary weight connections and redundant neurons from the initially assumed neural network structure. According to [20,30], the sparsity of the learning algorithm was measured by using the number of weights whose absolute values are ≤0.0099 and ≤0.01, respectively. In this study, we have arbitrary chosen 0.00099 as a threshold value which is more less than the existing thresholds in literature.…”

Section: Hidden Neuron Selection Criterionmentioning

confidence: 99%

Feedforward Neural Networks with a Hidden Layer Regularization Method

Alemu

Zhao

2018

Symmetry

View full text Add to dashboard Cite

In this paper, we propose a group Lasso regularization term as a hidden layer regularization method for feedforward neural networks. Adding a group Lasso regularization term into the standard error function as a hidden layer regularization term is a fruitful approach to eliminate the redundant or unnecessary hidden layer neurons from the feedforward neural network structure. As a comparison, a popular Lasso regularization method is introduced into standard error function of the network. Our novel hidden layer regularization method can force a group of outgoing weights to become smaller during the training process and can eventually be removed after the training process. This means it can simplify the neural network structure and it minimizes the computational cost. Numerical simulations are provided by using K-fold cross-validation method with K = 5 to avoid overtraining and to select the best learning parameters. The numerical results show that our proposed hidden layer regularization method prunes more redundant hidden layer neurons consistently for each benchmark dataset without loss of accuracy. In contrast, the existing Lasso regularization method prunes only the redundant weights of the network, but it cannot prune any redundant hidden layer neurons.

show abstract

Section: Hidden Neuron Selection Criterionmentioning

confidence: 99%

Feedforward Neural Networks with a Hidden Layer Regularization Method

Alemu

Zhao

2018

Symmetry

View full text Add to dashboard Cite

show abstract

“…Even the L 0 , L 1 , and L 1/2 regularizers can generate sparse results, but only the single sparse weight can be selected. So, it is still a challenge for us to decide which neuron is redundant [25], [37]- [40]. As a meaningful extension of Lasso, Group Lasso mainly has to do with variable selection on groups.…”

Section: Introductionmentioning

confidence: 99%

Convergence of a Gradient-Based Learning Algorithm With Penalty for Ridge Polynomial Neural Networks

Fan

Peng

et al. 2021

IEEE Access

View full text Add to dashboard Cite

“…e pruning method starts with a large network structure and then prunes unimportant weights or nodes [15][16][17][18][19].…”

Section: Introductionmentioning

confidence: 99%

Smoothing L0 Regularization for Extreme Learning Machine

Fan

Liu

2020

Mathematical Problems in Engineering

View full text Add to dashboard Cite

Extreme learning machine (ELM) has been put forward for single hidden layer feedforward networks. Because of its powerful modeling ability and it needs less human intervention, the ELM algorithm has been used widely in both regression and classification experiments. However, in order to achieve required accuracy, it needs many more hidden nodes than is typically needed by the conventional neural networks. This paper considers a new efficient learning algorithm for ELM with smoothing L0 regularization. A novel algorithm updates weights in the direction along which the overall square error is reduced the most and then this new algorithm can sparse network structure very efficiently. The numerical experiments show that the ELM algorithm with smoothing L0 regularization has less hidden nodes but better generalization performance than original ELM and ELM with L1 regularization algorithms.

show abstract

Batch gradient training method with smoothing $$\boldsymbol{\ell}_{\bf 0}$$ ℓ 0 regularization for feedforward neural networks

Cited by 19 publications

References 33 publications

Feedforward Neural Networks with a Hidden Layer Regularization Method

Feedforward Neural Networks with a Hidden Layer Regularization Method

Convergence of a Gradient-Based Learning Algorithm With Penalty for Ridge Polynomial Neural Networks

Smoothing L0 Regularization for Extreme Learning Machine

Contact Info

Product

Resources

About