A new sensitivity-based pruning technique for feed-forward neural networks that improves generalization

Abstract

Multi-layer neural networks of the backpropagation type (MLP-networks) became a well-established tool used in various application areas. Reliable solutions require, however, also sufficient generalization capabilities of the formed networks and an easy interpretation of their function. These characteristics are strongly related to less sensitive networks with an optimized network structure.In this paper, we will introduce a new pruning technique called SCGSIR that is inspired by the fast method of scaled conjugate gradients (SCG) and sensitivity analysis. Network sensitivity inhibited during training impacts efficient optimization of network structure. Experiments performed so far yield promising results outperforming the reference techniques when considering both their ability to find networks with optimum architecture and improved generalization.

Learning Sparse Neural Networks via Sensitivity-Driven Regularization

“…In this work, we propose to selectively prune each network parameter using the knowledge of sen-sitivity. Engelbrecht et al [19] and Mrazova et al [20,21] previously proposed sensitivity-based strategies for learning sparse architectures. In their work, the sensitivity is however defined as the variation of the network output with respect to a variation of the network inputs.…”