Determination of the number of redundant hidden units in a three-layered feedforward neural network

Tamura, Shinichi; Tateishi, Machiko; Matumoto, Masanobu; Akita, Shingo

doi:10.1109/ijcnn.1993.713925

Cited by 10 publications

(5 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The relatively simple structure of the empirical Fisher matrix inspired additional approaches. For example, Tamura et al [1993] and Fletcher et al [1998] use singular value decomposition of the Fisher matrix to determine the ideal number of neurons in each hidden layer. Assuming that outputs are linearly activated, they use the rank of the resulting covariance matrix of maximum likelihood to compute the number of neurons in the compressed network.…”

Section: Selection Based On 2nd Order Taylor Expansion Of the Trainin...mentioning

confidence: 99%

Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks

Hoefler¹,

Alistarh²,

Ben-Nun³

et al. 2021

Preprint

View full text Add to dashboard Cite

The growing energy and performance costs of deep learning have driven the community to reduce the size of neural networks by selectively pruning components. Similarly to their biological counterparts, sparse networks generalize just as well, if not better than, the original dense networks. Sparsity can reduce the memory footprint of regular networks to fit mobile devices, as well as shorten training time for ever growing networks. In this paper, we survey prior work on sparsity in deep learning and provide an extensive tutorial of sparsification for both inference and training. We describe approaches to remove and add elements of neural networks, different training strategies to achieve model sparsity, and mechanisms to exploit sparsity in practice. Our work distills ideas from more than 300 research papers and provides guidance to practitioners who wish to utilize sparsity today, as well as to researchers whose goal is to push the frontier forward. We include the necessary background on mathematical methods in sparsification, describe phenomena such as early structure adaptation, the intricate relations between sparsity and the training process, and show techniques for achieving acceleration on real hardware. We also define a metric of pruned parameter efficiency that could serve as a baseline for comparison of different sparse networks. We close by speculating on how sparsity can improve future workloads and outline major open problems in the field.. The supreme goal of all theory is to make the irreducible basic elements as simple and as few as possible without having to surrender the adequate representation of a single datum of experience -Albert Einstein, 1933 INTRODUCTIONDeep learning shows unparalleled promise for solving very complex real-world problems in areas such as computer vision, natural language processing, knowledge representation, recommendation systems, drug discovery, and many more. With this development, the field of machine learning is moving from traditional feature engineering to neural architecture engineering. However, still little is known about how to pick the right architecture to solve a specific task. Several methods such as translational equivariance in convolutional layers, recurrence, structured weight sharing, pooling, or locality are used to introduce strong inductive biases in the model design. Yet, the exact model size and capacity required for a task remain unknown and a common strategy is to train overparameterized models and compress them into smaller representations. Biological brains, especially the human brain, are hierarchical, sparse, and recurrent structures [Friston 2008] and one can draw some similarities with the inductive biases in today's artificial neural networks. Sparsity plays an important role in scaling biological brains-the more

show abstract

Section: Selection Based On 2nd Order Taylor Expansion Of the Trainin...mentioning

confidence: 99%

Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks

Hoefler¹,

Alistarh²,

Ben-Nun³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…The layers between have a configurable number of neurons, and each neuron has direct connections to all neurons of the subsequent layer. In general, the hidden layer determines the reliability of the model and the number of neurons is usually equal to the number of inputs [3].…”

Section: B Neural Network Estimationmentioning

confidence: 99%

“…In [2], when estimating the power consumed by the ARM big.LITTLE platform, the internal activity is supposed to be constant. In [3], a few performance events were manually selected based on a previous knowledge about the system behavior, to build a simple linear model. In [4], the authors present a power model based on dynamic and static contributions, but there is no mention on the impact of the external temperature.…”

Section: Introductionmentioning

confidence: 99%

Adaptive Power monitoring for self-aware embedded systems

Ahmad

Najem

Benoit

et al. 2015

2015 Nordic Circuits and Systems Conference (NORCAS): NORCHIP &Amp; International Symposium on System-on-Chip (SoC)

View full text Add to dashboard Cite

Dynamic Thermal and Power Management methods require efficient monitoring techniques. Based on a set of data collected by sensors, embedded models estimate online the power consumption: this task is a real challenge, since models must be both accurate and low cost, but they also have to be robust to variations. In this paper, we investigate a self-aware approach using the performance events and the external temperature. We present a solution (PESel) for the selection of the relevant information. This method is two times faster than existing solutions and provides better results compared to related works. The power models achieve a 96% accuracy with a temporal resolution of 100 ms, and negligible performance/energy overheads (less than 1%). Moreover, we show that our estimations are not sensitive to external temperature variations.

show abstract

“…In the mutual information based methods [33,34], singular value decomposition is used to analyze the hidden unit activation covariance matrix and the rank of the covariance matrix determines the optimal number of hidden units. Xing-Hu [35] suggests a two phase construction approach for pruning both input and hidden units of MLPs based on mutual information.…”

Section: Network Pruning and Related Workmentioning

confidence: 99%

A Novel Pruning Algorithm for Optimizing Feedforward Neural Network of Classification Problems

Augasta¹,

Kathirvalavakumar²

2011

Neural Process Lett

View full text Add to dashboard Cite

Optimizing the structure of neural networks is an essential step for the discovery of knowledge from data. This paper deals with a new approach which determines the insignificant input and hidden neurons to detect the optimum structure of a feedforward neural network. The proposed pruning algorithm, called as neural network pruning by significance (N2PS), is based on a new significant measure which is calculated by the Sigmoidal activation value of the node and all the weights of its outgoing connections. It considers all the nodes with significance value below the threshold as insignificant and eliminates them. The advantages of this approach are illustrated by implementing it on six different real datasets namely iris, breast-cancer, hepatitis, diabetes, ionosphere and wave. The results show that the proposed algorithm is quite efficient in pruning the significant number of neurons on the neural network models without sacrificing the networks performance.

show abstract

Determination of the number of redundant hidden units in a three-layered feedforward neural network

Cited by 10 publications

References 3 publications

Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks

Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks

Adaptive Power monitoring for self-aware embedded systems

A Novel Pruning Algorithm for Optimizing Feedforward Neural Network of Classification Problems

Contact Info

Product

Resources

About