Pruned and Structurally Sparse Neural Networks

Abstract

Advances in designing and training deep neural networks have led to the principle that the large and deeper a network is, the better it can perform. As a result, computational resources have become a key limiting factor in achieving better performance. One strategy to improve network capabilities while decreasing computation required is to replace dense fullyconnected and convolutional layers with sparse layers. In this paper we experiment with training on sparse neural network topologies. First, we test pruning-based sparse topologies, which use a network topology obtained by initially training a dense network and then pruning low-weight connections. Second, we test RadiX-Nets, a class of sparse network structures with proven connectivity and sparsity properties. Results show that compared to dense topologies, sparse structures show promise in training potential but also can exhibit highly nonlinear convergence, which merits further study.

Network Horizon Dynamics I: Qualitative Aspects

“…Networks are trained to produce desired outputs by adjusting their weights via numerical methods such as gradient descent. Training induces functional suppression of insignificant or redundant edges, which may be supplemented by explicit pruning [32,38]. Such processes are analogous to biological synaptic pruning, an important aspect of maturation [33,34].…”

“…In particular, hybrid local/random networks constructed by augmenting convolutional neural networks [35,36,37] with sparse random structure exhibit superior connectivity properties at reduced computational cost. Novel pseudorandom designs have already eclipsed standard architectures in accuracy and efficiency [38,39,40]. Such methods may allow construction of networks capable of next-generation tasks such as recognition of individuals among a large population, while democratizing access to state-of-the art technology.…”

“…However, many other architectures are possible, including RNNs and even networks of networks [77]. Absolute connectivity properties of DNNs have traditionally remained fixed following design, though subsequent alterations such as pruning have recently become common [78,32,33,34,38]. However, functional connectivity is determined not only by network topology, but also by edge weights, which change as the network learns.…”

Combinatorial Tiling for Sparse Neural Networks

“…Sparsely-connected neural networks exhibit lower computational complexity and lower memory requirements compared to their dense counterparts. They may originate by pruning a dense network as in the Banded Sparse Neural Networks [3], or result from training a fixed sparse topology as in the RadiX-Net [4]. The input data matrix may also be sparse, due to feature extraction techniques generating sparse representations (from, e.g., image, video, or signal data), or because input may be naturally sparse (e.g., graph inputs).…”

“…Images are interpolated to the number of neurons in the neural networks: 1024, 4096, 16384, and 65536. Several deep sparse neural networks are generated using RadiX-Net [4], with the number of neurons, layers, and bytes in Table I. This size, in bytes, assumes Compressed Row Storage (CRS) using four-byte values and indices.…”