LOss-Based SensiTivity rEgulaRization: Towards deep sparse neural networks

Tartaglione, Enzo; Bragagnolo, Andrea; Fiandrotti, Attilio; Grangetto, Marco

doi:10.1016/j.neunet.2021.11.029

Cited by 22 publications

(16 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An effective way to avoid this issue is presented in [17], where the idea of output-based sensitivity is introduced: weights are selectively penalized depending on their capability to induce variations on network outputs when changed. A refinement of this method is presented in [18], where state-ofthe-art results are reached in image classification thanks to the introduction of loss-based sensitivity, which aims at shrinking weights which are contributing the least to the final loss value.…”

Section: Related Workmentioning

confidence: 99%

“…A drawback of these methods is that neural weights' norms are all driven close to zero without taking account of weight relevance in the neural architecture, as discussed in detail in [17,18].…”

Section: Introductionmentioning

confidence: 99%

“…It allows to derive a new weights' update rule which selectively decreases the norm of non relevant weights, while performing a classic update for relevant ones. Weights' update directly follows from loss optimization, not requiring the definition of ad hoc update rules, as frequently done (see [14,[17][18][19]). Shrunk weights are then pruned in order to sparsify the neural architecture.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Regularization-based Pruning of Irrelevant Weights in Deep Neural Architectures

Bonetta¹,

Ribero²,

Cancelliere³

2022

Preprint

View full text Add to dashboard Cite

Deep neural networks exploiting millions of parameters are nowadays the norm in deep learning applications. This is a potential issue because of the great amount of computational resources needed for training, and of the possible loss of generalization performance of overparametrized networks. We propose in this paper a method for learning sparse neural topologies via a regularization technique which identifies non relevant weights and selectively shrinks their norm, while performing a classic update for relevant ones. This technique, which is an improvement of classical weight decay, is based on the definition of a regularization term which can be added to any loss functional regardless of its form, resulting in a unified general framework exploitable in many different contexts. The actual elimination of parameters identified as irrelevant is handled by an iterative pruning algorithm. We tested the proposed technique on different image classification and Natural language generation tasks, obtaining results on par or better then competitors in terms of sparsity and metrics, while achieving strong models compression.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Regularization-based Pruning of Irrelevant Weights in Deep Neural Architectures

Bonetta¹,

Ribero²,

Cancelliere³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…A number of approaches have been proposed, especially in the recent years [10,12,15]. One recent state-of-the-art approach, LOBSTER [23] proposes to penalize the parameters by their gradient-weighted 2 norm, leading to the update rule…”

Section: Enforcing Rem With Pruningmentioning

confidence: 99%

REM: Routing Entropy Minimization for Capsule Networks

Renzulli¹,

Tartaglione²,

Grangetto³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Capsule Networks ambition is to build an explainable and biologically-inspired neural network model. One of their main innovations relies on the routing mechanism which extracts a parse tree: its main purpose is to explicitly build relationships between capsules. However, their true potential in terms of explainability has not surfaced yet: these relationships are extremely heterogeneous and difficult to understand. This paper proposes REM, a technique which minimizes the entropy of the parse tree-like structure, improving its explainability. We accomplish this by driving the model parameters distribution towards low entropy configurations, using a pruning mechanism as a proxy. We also generate static parse trees with no performance loss, showing that, with REM, Capsule Networks build stronger relationships between capsules.

show abstract

“…Historically techniques based on the L2 regularization are among the most popular ways to achieve highly sparsifying models. A central drawback of such algorithms is that they do not directly account for weight relevance in the neural architecture (see [3,4]), but the entire set of parameters is forced towards very small values.…”

Section: Introductionmentioning

confidence: 99%

Lightening CNN architectures by regularization driven weights' pruning

Bonetta¹,

Cancelliere²

2022

ESANN 2022 Proceedings

View full text Add to dashboard Cite

Deep learning models are getting increasingly big, leading towards overparametrized architectures with high computational and storage requirements. This hinders the possibility to train/deploy them on IoT or mobile devices, while also creating concerns about their environmental fingerprint. We propose a regularization technique which allows to selectively shrink the norm of non significant weights in order to subsequently prune them, generating highly compressed models. We tested the proposed technique on three well known image classification tasks, obtaining results on par or better than competitors in terms of sparsity and metrics.

show abstract

LOss-Based SensiTivity rEgulaRization: Towards deep sparse neural networks

Cited by 22 publications

References 15 publications

Regularization-based Pruning of Irrelevant Weights in Deep Neural Architectures

Regularization-based Pruning of Irrelevant Weights in Deep Neural Architectures

REM: Routing Entropy Minimization for Capsule Networks

Lightening CNN architectures by regularization driven weights' pruning

Contact Info

Product

Resources

About