Adam Induces Implicit Weight Sparsity in Rectifier Neural Networks

Yaguchi, Atsushi; Suzuki, Taiji; Asano, Wataru; Nitta, Shuhei; Sakata, Yoshihito; Tanizawa, Akiyuki

doi:10.48550/arxiv.1812.08119

Cited by 2 publications

(6 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As an alternative to SGD we also evaluated our regularization strategy against Adam optimization (Table 2). In alignment with the reported tendency of this adaptive method [11], our visualization clearly indicates high sparsity (Figure A.1). We further demonstrate that sparsity regularization can cause strongly correlating features if applied to method-induced sparsity (Figure A.2).…”

Section: Targeted Sparsity Regularizationsupporting

confidence: 86%

“…Given inappropriate combinations of network size and training data special measures have to be considered to prevent overfitting [8]. Popular countermeasures include (i) artificially increasing the amount of training data [9,10]; (ii) reducing the network's capacity; and (iii) changing the learning strategy [11]. The reduction of the capacity can either be done explicitly by reducing the amount of learnable parameters (i.e.…”

Section: Introductionmentioning

confidence: 99%

“…Based on the observation that large deep neural networks often lead to redundancies [25], many of the activation-/ weight-based strategies aim to extenuate the model complexity by using weight decay [15,26] or pruning the network directly [27,13]. Similarly, it has been shown that classical L1 regularization induces sparsity while L2 regularization causes the decay of weights over time [11,28]. Even though these techniques can improve the generalizability and performance of the network, they underutilize the potential capacity of the model [29].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Exploiting the Full Capacity of Deep Neural Networks while Avoiding Overfitting by Targeted Sparsity Regularization

Huesmann,

Klemm,

Linsen

et al. 2020

Preprint

View full text Add to dashboard Cite

Overfitting is one of the most common problems when training deep neural networks on comparatively small datasets. Here, we demonstrate that neural network activation sparsity is a reliable indicator for overfitting which we utilize to propose novel targeted sparsity visualization and regularization strategies. Based on these strategies we are able to understand and counteract overfitting caused by activation sparsity and filter correlation in a targeted layer-by-layer manner. Our results demonstrate that targeted sparsity regularization can efficiently be used to regularize well-known datasets and architectures with a significant increase in image classification performance while outperforming both dropout and batch normalization. Ultimately, our study reveals novel insights into the contradicting concepts of activation sparsity and network capacity by demonstrating that targeted sparsity regularization enables salient and discriminative feature learning while exploiting the full capacity of deep models without suffering from overfitting, even when trained excessively.

show abstract

Section: Targeted Sparsity Regularizationsupporting

confidence: 86%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Exploiting the Full Capacity of Deep Neural Networks while Avoiding Overfitting by Targeted Sparsity Regularization

Huesmann,

Klemm,

Linsen

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…With the finding of increased sparsity due to BN + ReLU, a natural question is whether the sparsity is favorable for the network. Several recent works treated the sparsity as a windfall in network training since it automatically generates a pruned network [20,21]. However, in this work, we show that the accuracy of the sparsified network only matches that using uniform channel pruning, suggesting detrimental effects of the collapsed filters.…”

Section: Introductionmentioning

confidence: 58%

“…Several recent studies have investigated the dying probability with network depth [25]. The ReLU-related network sparsity in neural networks has also been noticed by two contemporary works [21,20], yet with no special attention to batch normalization. The sparsification change with different hyper-parameters such as LR, weight decay and training algorithms was experimentally analyzed in [21].…”

Section: Dying Relumentioning

confidence: 98%

How Does BN Increase Collapsed Neural Network Filters?

Zhou¹,

Wang²,

Luo³

et al. 2020

Preprint

View full text Add to dashboard Cite

Improving sparsity of deep neural networks (DNNs) is essential for network compression and has drawn much attention. In this work, we disclose a harmful sparsifying process called filter collapse, which is common in DNNs with batch normalization (BN) and rectified linear activation functions (e.g. ReLU, Leaky ReLU). It occurs even without explicit sparsity-inducing regularizations such as L 1 . This phenomenon is caused by the normalization effect of BN, which induces a non-trainable region in the parameter space and reduces the network capacity as a result. This phenomenon becomes more prominent when the network is trained with large learning rates (LR) or adaptive LR schedulers, and when the network is finetuned. We analytically prove that the parameters of BN tend to become sparser during SGD updates with high gradient noise and that the sparsifying probability is proportional to the square of learning rate and inversely proportional to the square of the scale parameter of BN. To prevent the undesirable collapsed filters, we propose a simple yet effective approach named post-shifted BN (psBN), which has the same representation ability as BN while being able to automatically make BN parameters trainable again as they saturate during training. With psBN, we can recover collapsed filters and increase the model performance in various tasks such as classification on CIFAR-10 and object detection on MS-COCO2017.

show abstract

Adam Induces Implicit Weight Sparsity in Rectifier Neural Networks

Cited by 2 publications

References 9 publications

Exploiting the Full Capacity of Deep Neural Networks while Avoiding Overfitting by Targeted Sparsity Regularization

Exploiting the Full Capacity of Deep Neural Networks while Avoiding Overfitting by Targeted Sparsity Regularization

How Does BN Increase Collapsed Neural Network Filters?

Contact Info

Product

Resources

About