SeReNe: Sensitivity-Based Regularization of Neurons for Structured Sparsity in Neural Networks

Tartaglione, Enzo; Bragagnolo, Andrea; Odierna, Francesco; Fiandrotti, Attilio; Grangetto, Marco

doi:10.1109/tnnls.2021.3084527

Cited by 14 publications

(6 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…From this, we observe that reducing the complexity of the backbone would result in the overall reduction of the complexity for the entire model N , and towards this end pruning has already proved to be an effective approach [9,10]. Pruning approaches can be divided into two groups.…”

Section: Effect Of Pruned Backbones To Capsule Layersmentioning

confidence: 92%

“…Unstructured pruning methods aim at minimizing the cardinality ∥w∥ 0 of the parameters in the model, regardless the output topology [16,17,18]. On the other hand, structured approaches drop groups of weight connections entirely, such as entire channels or filters, imposing a regular pruned topology [9,19]. As an effect, they minimize the cardinality of some i-th intermediate layer's output ∥x i ∥ 0 .…”

Section: Effect Of Pruned Backbones To Capsule Layersmentioning

confidence: 99%

“…However, the problem of over-parametrization and the difficulty of deploying networks in scenarios with limited memory and resources is not peculiar only to CapsNets, but also to CNNs. To this end, pruning methods (which reduce the size and complexity of a neural network model) have gained more and more attention lately [9]. Learning sparse topologies can effectively reduce the network footprint and speedup execution with negligible performance loss [10].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Towards Efficient Capsule Networks

Renzulli

Grangetto

2022

2022 IEEE International Conference on Image Processing (ICIP)

View full text Add to dashboard Cite

From the moment Neural Networks dominated the scene for image processing, the computational complexity needed to solve the targeted tasks skyrocketed: against such an unsustainable trend, many strategies have been developed, ambitiously targeting performance's preservation. Promoting sparse topologies, for example, allows the deployment of deep neural networks models on embedded, resourceconstrained devices. Recently, Capsule Networks were introduced to enhance explainability of a model, where each capsule is an explicit representation of an object or its parts. These models show promising results on toy datasets, but their low scalability prevents deployment on more complex tasks. In this work, we explore sparsity besides capsule representations to improve their computational efficiency by reducing the number of capsules. We show how pruning with Capsule Network achieves high generalization with less memory requirements, computational effort, and inference and training time.

show abstract

Section: Effect Of Pruned Backbones To Capsule Layersmentioning

confidence: 92%

Section: Effect Of Pruned Backbones To Capsule Layersmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Towards Efficient Capsule Networks

Renzulli

Grangetto

2022

2022 IEEE International Conference on Image Processing (ICIP)

View full text Add to dashboard Cite

show abstract

“…A recent work [18] even showed that structured procedures greatly benefit endto-end compression and the deployment on embedded devices. Following this trend, for our experiments, we used the structured pruning strategy described in [19].…”

Section: Methodsmentioning

confidence: 99%

Efficient Inference Of Image-Based Neural Network Models In Reconfigurable Systems With Pruning And Quantization

Flich

Medina

Catalan

et al. 2022

2022 IEEE International Conference on Image Processing (ICIP)

View full text Add to dashboard Cite

Neural networks (NN) for image processing in embedded systems expose two conflicting requirements: increasing computing power needs as models become more complex and constrained resource budget. In order to alleviate this problems, model compression based on quantization and pruning techniques are common. Derived models then need to fit on reconfigurable systems such as FPGAs for the embedded system to work properly. In this paper, we present HLSinf, an open source framework for the development of custom NN accelerators for FPGAs which provides efficient support to quantized and pruned NN models. With HLSinf, significant inference speedups can be obtained for typical medical image-based applications. In particular, we obtain up to 90x speedup factor when we combine quantization/pruning with the flexibility of HLSinf compared to CPU.

show abstract

“…Can they be determined earlier, during a normal vanilla training? deployment time and making inference more efficient [7,8,9,10,11]. A recent work, the lottery ticket hypothesis [12], suggests that the fate of a parameter, namely whether it is useful for training (winner at the lottery of initialization) or if it can be removed from the architecture, is decided already at the initialization step.…”

Section: Vanilla Trainingmentioning

confidence: 99%

The Rise of the Lottery Heroes: Why Zero-Shot Pruning is Hard

Tartaglione

2022

2022 IEEE International Conference on Image Processing (ICIP)

View full text Add to dashboard Cite

Recent advances in deep learning optimization showed that just a subset of parameters are really necessary to successfully train a model. Potentially, such a discovery has broad impact from the theory to application; however, it is known that finding these trainable sub-network is a typically costly process. This inhibits practical applications: can the learned sub-graph structures in deep learning models be found at training time? In this work we explore such a possibility, observing and motivating why common approaches typically fail in the extreme scenarios of interest, and proposing an approach which potentially enables training with reduced computational effort. The experiments on either challenging architectures and datasets suggest the algorithmic accessibility over such a computational gain, and in particular a trade-off between accuracy achieved and training complexity deployed emerges.

show abstract

SeReNe: Sensitivity-Based Regularization of Neurons for Structured Sparsity in Neural Networks

Cited by 14 publications

References 12 publications

Towards Efficient Capsule Networks

Towards Efficient Capsule Networks

Efficient Inference Of Image-Based Neural Network Models In Reconfigurable Systems With Pruning And Quantization

The Rise of the Lottery Heroes: Why Zero-Shot Pruning is Hard

Contact Info

Product

Resources

About