Pruning Artificial Neural Networks: A Way to Find Well-Generalizing, High-Entropy Sharp Minima

Tartaglione, Enzo; Bragagnolo, Andrea; Grangetto, Marco

doi:10.1007/978-3-030-61616-8_6

Cited by 11 publications

(5 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Furthermore, Han et al observed such a phenomenon for unstructured pruning [16]. Recently, it has been showed that the average entropy in the bottleneck layer for pruned backbones (in our case, x B ) is higher than in non-pruned ones [21]: this results in the propagation of less specific and more general information, which prevents features overfit, on top of which capsules layers can extract much more accurate information.…”

Section: Resultsmentioning

confidence: 61%

Towards Efficient Capsule Networks

Renzulli

Grangetto

2022

2022 IEEE International Conference on Image Processing (ICIP)

View full text Add to dashboard Cite

From the moment Neural Networks dominated the scene for image processing, the computational complexity needed to solve the targeted tasks skyrocketed: against such an unsustainable trend, many strategies have been developed, ambitiously targeting performance's preservation. Promoting sparse topologies, for example, allows the deployment of deep neural networks models on embedded, resourceconstrained devices. Recently, Capsule Networks were introduced to enhance explainability of a model, where each capsule is an explicit representation of an object or its parts. These models show promising results on toy datasets, but their low scalability prevents deployment on more complex tasks. In this work, we explore sparsity besides capsule representations to improve their computational efficiency by reducing the number of capsules. We show how pruning with Capsule Network achieves high generalization with less memory requirements, computational effort, and inference and training time.

show abstract

Section: Resultsmentioning

confidence: 61%

Towards Efficient Capsule Networks

Renzulli

Grangetto

2022

2022 IEEE International Conference on Image Processing (ICIP)

View full text Add to dashboard Cite

show abstract

“…The lottery ticket hypothesis. It is a known fact that deep neural networks are typically over-parametrized and, after training, a part of the parameters can be removed without harming the performance, or even slightly improving the performance in small pruning regimes [7,13]. However, an interesting question rises: is it possible to train a subnetwork, still achieving the same performance as training the full model?…”

Section: The Lottery Of the Initializationmentioning

confidence: 99%

The Rise of the Lottery Heroes: Why Zero-Shot Pruning is Hard

Tartaglione

2022

2022 IEEE International Conference on Image Processing (ICIP)

View full text Add to dashboard Cite

Recent advances in deep learning optimization showed that just a subset of parameters are really necessary to successfully train a model. Potentially, such a discovery has broad impact from the theory to application; however, it is known that finding these trainable sub-network is a typically costly process. This inhibits practical applications: can the learned sub-graph structures in deep learning models be found at training time? In this work we explore such a possibility, observing and motivating why common approaches typically fail in the extreme scenarios of interest, and proposing an approach which potentially enables training with reduced computational effort. The experiments on either challenging architectures and datasets suggest the algorithmic accessibility over such a computational gain, and in particular a trade-off between accuracy achieved and training complexity deployed emerges.

show abstract

“…Many techniques have been proposed recently to reduce such complexity [15][16][17]. These approaches include the so-called pruning techniques, whose aim is to detect and remove the irrelevant parameters from a model [18].…”

Section: The European Distributed Deep Learning Librarymentioning

confidence: 99%

The DeepHealth Toolkit: A Key European Free and Open-Source Software for Deep Learning and Computer Vision Ready to Exploit Heterogeneous HPC and Cloud Architectures

Aldinucci

Atienza

Bolelli

et al. 2022

Technologies and Applications for Big Data Value

Self Cite

View full text Add to dashboard Cite

At the present time, we are immersed in the convergence between Big Data, High-Performance Computing and Artificial Intelligence. Technological progress in these three areas has accelerated in recent years, forcing different players like software companies and stakeholders to move quickly. The European Union is dedicating a lot of resources to maintain its relevant position in this scenario, funding projects to implement large-scale pilot testbeds that combine the latest advances in Artificial Intelligence, High-Performance Computing, Cloud and Big Data technologies. The DeepHealth project is an example focused on the health sector whose main outcome is the DeepHealth toolkit, a European unified framework that offers deep learning and computer vision capabilities, completely adapted to exploit underlying heterogeneous High-Performance Computing, Big Data and cloud architectures, and ready to be integrated into any software platform to facilitate the development and deployment of new applications for specific problems in any sector. This toolkit is intended to be one of the European contributions to the field of AI. This chapter introduces the toolkit with its main components and complementary tools, providing a clear view to facilitate and encourage its adoption and wide use by the European community of developers of AI-based solutions and data scientists working in the healthcare sector and others.

show abstract

Pruning Artificial Neural Networks: A Way to Find Well-Generalizing, High-Entropy Sharp Minima

Cited by 11 publications

References 4 publications

Towards Efficient Capsule Networks

Towards Efficient Capsule Networks

The Rise of the Lottery Heroes: Why Zero-Shot Pruning is Hard

The DeepHealth Toolkit: A Key European Free and Open-Source Software for Deep Learning and Computer Vision Ready to Exploit Heterogeneous HPC and Cloud Architectures

Contact Info

Product

Resources

About