What is the State of Neural Network Pruning?

Blalock, Davis; Ortiz, Jose Javier Gonzalez; Frankle, Jonathan; Guttag, John V.

doi:10.48550/arxiv.2003.03033

Cited by 135 publications

(216 citation statements)

References 0 publications

Supporting

Mentioning

207

Contrasting

Unclassified

Order By: Relevance

“…This is due to the long time it takes to train a neural network, large data sets, and large number of parameters in them. Some recent studies [1,2,3] showed that many weights are excessive and can be removed without loss (or with a small loss) of neural network performance.…”

Section: Intorductionmentioning

confidence: 99%

Neural Network Optimization for Reinforcement Learning Tasks Using Sparse Computations

Ivanov¹,

Киселев²,

Larionov³

2022

Preprint

View full text Add to dashboard Cite

This article proposes a sparse computation-based method for optimizing neural networks for reinforcement learning (RL) tasks. This method combines two ideas: neural network pruning and taking into account input data correlations; it makes it possible to update neuron states only when changes in them exceed a certain threshold. It significantly reduces the number of multiplications when running neural networks. We tested different RL tasks and achieved 20-150x reduction in the number of multiplications. There were no substantial performance losses; sometimes the performance even improved.

show abstract

Section: Intorductionmentioning

confidence: 99%

Neural Network Optimization for Reinforcement Learning Tasks Using Sparse Computations

Ivanov¹,

Киселев²,

Larionov³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…After the attack, some neurons of the poisoned model contained the adverse effect. The intuition of Pruning [120] is to remove those for sanitizing the model. The authors in [121] applied Pruning to an FL environment, successfully preventing Poisoning attacks.…”

Section: B Defending Integrity and Availabilitymentioning

confidence: 99%

On the Security & Privacy in Federated Learning

Abad¹,

Picek²,

Ramírez-Durán³

et al. 2021

Preprint

View full text Add to dashboard Cite

Advances in Machine Learning (ML) and its wide range of applications boosted its popularity. Recent privacy awareness initiatives as the EU General Data Protection Regulation (GDPR) -European Parliament and Council Regulation No 2016/679, subdued ML to privacy and security assessments. Federated Learning (FL) grants a privacy-driven, decentralized training scheme that improves ML models' security. The industry's fast-growing adaptation and security evaluations of FL technology exposed various vulnerabilities. Depending on the FL phase, i.e., training or inference, the adversarial actor capabilities, and the attack type threaten FL's confidentiality, integrity, or availability (CIA). Therefore, the researchers apply the knowledge from distinct domains as countermeasures, like cryptography and statistics.This work assesses the CIA of FL by reviewing the state-of-theart (SoTA) for creating a threat model that embraces the attack's surface, adversarial actors, capabilities, and goals. We propose the first unifying taxonomy for attacks and defenses by applying this model. Additionally, we provide critical insights extracted by applying the suggested novel taxonomies to the SoTA, yielding promising future research directions.

show abstract

“…Specifically, if the sparsity pattern obtained by iterative magnitude-based pruning is applied to the NN with all non-pruned weights reset to their initial values, the resulting sparse NN can be trained to achieve the same or even better performance as the original dense one. A multitude of follow-up works explore this research direction (see the recent review article [15] for a list of references). However, how to find a method to obtain a working sparsity pattern for a NN without first training the dense NN remains an open research question.…”

Section: A Pruning and Sparsitymentioning

confidence: 99%

Pruning a restricted Boltzmann machine for quantum state reconstruction

Golubeva,

Melko

2021

Preprint

View full text Add to dashboard Cite

Restricted Boltzmann machines (RBMs) have proven to be a powerful tool for learning quantum wavefunction representations from qubit projective measurement data. Since the number of classical parameters needed to encode a quantum wavefunction scales rapidly with the number of qubits, the ability to learn efficient representations is of critical importance. In this paper we study magnitudebased pruning as a way to compress the wavefunction representation in an RBM, focusing on RBMs trained on data from the transverse-field Ising model in one dimension. We find that pruning can reduce the total number of RBM weights, but the threshold at which the reconstruction accuracy starts to degrade varies significantly depending on the phase of the model. In a gapped region of the phase diagram, the RBM admits pruning over half of the weights while still accurately reproducing relevant physical observables. At the quantum critical point however, even a small amount of pruning can lead to significant loss of accuracy in the physical properties of the reconstructed quantum state. Our results highlight the importance of tracking all relevant observables as their sensitivity varies strongly with pruning. Finally, we find that sparse RBMs are trainable and discuss how a successful sparsity pattern can be created without pruning.

show abstract

What is the State of Neural Network Pruning?

Cited by 135 publications

References 0 publications

Neural Network Optimization for Reinforcement Learning Tasks Using Sparse Computations

Neural Network Optimization for Reinforcement Learning Tasks Using Sparse Computations

On the Security & Privacy in Federated Learning

Pruning a restricted Boltzmann machine for quantum state reconstruction

Contact Info

Product

Resources

About