Neuron Pruning for Compressing Deep Networks Using Maxout Architectures

Rueda, Fernando Moya; Grzeszick, René; Fink, Gernot A.

doi:10.1007/978-3-319-66709-6_15

Cited by 10 publications

(3 citation statements)

References 8 publications

(21 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It is essential to do a trade-off between the compressed rate and the accuracy reduction. Current popular methods include pruning [14] [15] [16], quantization [17], parameter sharing [18], knowledge distillation, lowrank approximation and direct design of compact models, etc. Besides, another similar method named Neural Architecture Search (NAS) is also widely considered to explore a suitable lightweight model for resource-limited devices.…”

Section: A Model Compressionmentioning

confidence: 99%

Towards Fairer and More Efficient Federated Learning via Multidimensional Personalized Edge Models

Wang¹,

Guo²,

Zhang³

et al. 2023

Preprint

View full text Add to dashboard Cite

Federated learning (FL) is an emerging technique that trains massive and geographically distributed edge data while maintaining privacy. However, FL has inherent challenges in terms of fairness and computational efficiency due to the rising heterogeneity of edges, and thus usually result in suboptimal performance in recent state-of-the-art (SOTA) solutions. In this paper, we propose a Customized Federated Learning (CFL) system to eliminate FL heterogeneity from multiple dimensions. Specifically, CFL tailors personalized models from the specially designed global model for each client, jointly guided an online trained model-search helper and a novel aggregation algorithm. Extensive experiments demonstrate that CFL has full-stack advantages for both FL training and edge reasoning and significantly improves the SOTA performance w.r.t. model accuracy (up to 7.2% in the non-heterogeneous environment and up to 21.8% in the heterogeneous environment), efficiency, and FL fairness.

show abstract

Section: A Model Compressionmentioning

confidence: 99%

Towards Fairer and More Efficient Federated Learning via Multidimensional Personalized Edge Models

Wang¹,

Guo²,

Zhang³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Thus, it provides speedup and energy reduction; (2) Neuron pruning eliminates the entire rows/columns in the weight matrices reducing the weight matrices' dimensions proportionally, which could be efficiently implemented in the hardware compared to unstructured weight pruning [9]; (3) It also provides a way to determine the optimal number of neurons for a given network architecture [10]. Accordingly, many works have proposed various approaches to implement neuron pruning for the pursuit of a balance between compression ratio and accuracy [10][11][12][13][14].…”

Section: Introductionmentioning

confidence: 99%

Towards Efficient Neuromorphic Hardware: Unsupervised Adaptive Neuron Pruning

et al. 2020

View full text Add to dashboard Cite

To solve real-time challenges, neuromorphic systems generally require deep and complex network structures. Thus, it is crucial to search for effective solutions that can reduce network complexity, improve energy efficiency, and maintain high accuracy. To this end, we propose unsupervised pruning strategies that are focused on pruning neurons while training in spiking neural networks (SNNs) by utilizing network dynamics. The importance of neurons is determined by the fact that neurons that fire more spikes contribute more to network performance. Based on these criteria, we demonstrate that pruning with an adaptive spike count threshold provides a simple and effective approach that can reduce network size significantly and maintain high classification accuracy. The online adaptive pruning shows potential for developing energy-efficient training techniques due to less memory access and less weight-update computation. Furthermore, a parallel digital implementation scheme is proposed to implement spiking neural networks (SNNs) on field programmable gate array (FPGA). Notably, our proposed pruning strategies preserve the dense format of weight matrices, so the implementation architecture remains the same after network compression. The adaptive pruning strategy enables 2.3× reduction in memory size and 2.8× improvement on energy efficiency when 400 neurons are pruned from an 800-neuron network, while the loss of classification accuracy is 1.69%. And the best choice of pruning percentage depends on the trade-off among accuracy, memory, and energy. Therefore, this work offers a promising solution for effective network compression and energy-efficient hardware implementation of neuromorphic systems in real-time applications.

show abstract

“…In related work, [132] has applied AdaBoost to increase the resiliency of the overall system but has not explored in the energy reduction perspective. Moreover, the idea of DNS is completely different from neurons and weights pruning [133] performed during the training phase of the network to reduce the number of redundant neurons and parameters. Nevertheless, we can apply such methods to reduce each BL size.…”

Section: ) Approximationmentioning

confidence: 99%

Edge processing in IoT using approximate and in-memory computing

Bose¹

View full text Add to dashboard Cite

First and foremost, I am grateful and would like to express my sincere gratitude to my supervisor, Prof. Arindam Basu, for his continuous guidance, support, and suggestion throughout my Ph.D. in NTU. Without his vision and faith in me, I could not have come this far. He has been very supportive and had given me the freedom to explore various research directions from day one of my Ph.D. I am amazed and inspired by his immense inter-disciplinary knowledge and passion for research. He not only improvised my problem-solving skills but also taught me to communicate my thoughts to others concisely. I am incredibly fortunate to have him as a mentor in both academic and non-academic life. He is a person to look up to, and I am forever thankful to him for the opportunity he provided four years back and shaping my entire career.

show abstract

Neuron Pruning for Compressing Deep Networks Using Maxout Architectures

Cited by 10 publications

References 8 publications

Towards Fairer and More Efficient Federated Learning via Multidimensional Personalized Edge Models

Towards Fairer and More Efficient Federated Learning via Multidimensional Personalized Edge Models

Towards Efficient Neuromorphic Hardware: Unsupervised Adaptive Neuron Pruning

Edge processing in IoT using approximate and in-memory computing

Contact Info

Product

Resources

About