Weight smoothing to improve network generalization

Jean, Jack; Wang, Jin

doi:10.1109/72.317727

Cited by 37 publications

(14 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This concept has been explored in a variety of formulations, e.g. by means of weight decay [21], weight smoothing [19], label smoothing [38], or penalizing the norm of the output derivative with respect to the network weights [14]. Of particular interest to our problem are methods that regularize by penalizing the norm of the Jacobian with respect to the input [28,39].…”

Section: Background and Previous Workmentioning

confidence: 99%

Single-Frame Regularization for Temporally Stable CNNs

Eilertsen

Mantiuk

Unger

2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

Convolutional neural networks (CNNs) can model complicated non-linear relations between images. However, they are notoriously sensitive to small changes in the input. Most CNNs trained to describe image-to-image mappings generate temporally unstable results when applied to video sequences, leading to flickering artifacts and other inconsistencies over time. In order to use CNNs for video material, previous methods have relied on estimating dense frame-to-frame motion information (optical flow) in the training and/or the inference phase, or by exploring recurrent learning structures. We take a different approach to the problem, posing temporal stability as a regularization of the cost function. The regularization is formulated to account for different types of motion that can occur between frames, so that temporally stable CNNs can be trained without the need for video material or expensive motion estimation. The training can be performed as a fine-tuning operation, without architectural modifications of the CNN. Our evaluation shows that the training strategy leads to large improvements in temporal smoothness. Moreover, for small datasets the regularization can help in boosting the generalization performance to a much larger extent than what is possible with naïve augmentation strategies.

show abstract

Section: Background and Previous Workmentioning

confidence: 99%

Single-Frame Regularization for Temporally Stable CNNs

Eilertsen

Mantiuk

Unger

2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

show abstract

“…When the m.s.e. decreases, the adaptive learning rate Á.n/ increases according to (13). The enlarged Á.n/ accelerates the decrement of the error.…”

Section: Simulation Resultsmentioning

confidence: 99%

A Modified Error Function to Improve the Error Back-Propagation Algorithm for Multi-Layer Perceptrons

Oh¹,

Lee²

1995

ETRI J

View full text Add to dashboard Cite

“…To avoid the local minima, our training algorithm is gradient descent with momentum [9] and adaptive learning rate [10] for finding a set of weights which minimizes the MSE. And the weights' initial values are between -1 and 1 at random.…”

Section: Training Algorithmmentioning

confidence: 99%

Machine-Printed Traditional Mongolian Characters Recognition Using BP Neural Networks

Wei

Gao

2009

2009 International Conference on Computational Intelligence and Software Engineering

View full text Add to dashboard Cite

This paper presents a new method to recognize machine-printed traditional Mongolian characters by using back-propagation (BP) neural networks. First, the set of traditional Mongolian characters is divided into five subsets according to each character's position (initial, medial or final) within a word and some steady structural features. Then, each subset is trained and recognized by using a BP neural network with particular architecture. Thus, there are five BP neural networks in total, which can recognize all the traditional Mongolian characters. The experimental results are provided and show that the BP neural networks have good performance for the machine-printed traditional Mongolian characters recognition.

show abstract

Weight smoothing to improve network generalization

Cited by 37 publications

References 18 publications

Single-Frame Regularization for Temporally Stable CNNs

Single-Frame Regularization for Temporally Stable CNNs

A Modified Error Function to Improve the Error Back-Propagation Algorithm for Multi-Layer Perceptrons

Machine-Printed Traditional Mongolian Characters Recognition Using BP Neural Networks

Contact Info

Product

Resources

About