Sparsity Meets Robustness: Channel Pruning for the Feynman-Kac Formalism Principled Robust Deep Neural Nets

Dinh, Thu; Bao, Wang; Bertozzi, Andrea L.; Osher, Stanley; Xin, Jack

doi:10.1007/978-3-030-64580-9_31

Cited by 10 publications

(2 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The core matter is the evaluation of proximal operators [5]. The proximal algorithms are widely used in statistical computing and machine learning [42], channel pruning of neural networks [14,58], image processing [6], matrix completion [31,61], computational optimal transport [39,47], game theory and optimal control [3], etc. Proximal operators can be viewed as backward Euler method, see Section 2 for a brief mathematical introduction.…”

Section: More Related Workmentioning

confidence: 99%

Proximal Implicit ODE Solvers for Accelerating Learning Neural ODEs

Baker¹,

Xia²,

Wang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Learning neural ODEs often requires solving very stiff ODE systems, primarily using explicit adaptive step size ODE solvers. These solvers are computationally expensive, requiring the use of tiny step sizes for numerical stability and accuracy guarantees. This paper considers learning neural ODEs using implicit ODE solvers of different orders leveraging proximal operators. The proximal implicit solver consists of inner-outer iterations: the inner iterations approximate each implicit update step using a fast optimization algorithm, and the outer iterations solve the ODE system over time. The proximal implicit ODE solver guarantees superiority over explicit solvers in numerical stability and computational efficiency. We validate the advantages of proximal implicit solvers over existing popular neural ODE solvers on various challenging benchmark tasks, including learning continuous-depth graph neural networks and continuous normalizing flows.

show abstract

Section: More Related Workmentioning

confidence: 99%

Proximal Implicit ODE Solvers for Accelerating Learning Neural ODEs

Baker¹,

Xia²,

Wang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…An ideal pruning algorithm with promising fidelity should not incur a significant accuracy decline when compared with the original model. However, the discussion of the impact of pruning to measurements beyond fidelity, such as robustness, is still in its nascent phase [41,42,43]. As robustness is a representative property specification of a neural network model that concerns the security of its actual deployment, unveiling the influence of pruning on robustness could provide a guarantee to the trustworthiness of pruning techniques.…”

Section: Neural Network Pruningmentioning

confidence: 99%

Supervised Robustness-preserving Data-free Neural Network Pruning

Meng¹,

Bai²,

Teo³

et al. 2022

Preprint

View full text Add to dashboard Cite

When deploying pre-trained neural network models in real-world applications, model consumers often encounter resource-constraint platforms such as mobile and smart devices. They typically use the pruning technique to reduce the size and complexity of the model, generating a lighter one with less resource consumption. Nonetheless, most existing pruning methods are proposed with a premise that the model after being pruned has a chance to be fine-tuned or even retrained based on the original training data. This may be unrealistic in practice, as the data controllers are often reluctant to provide their model consumers with the original data. In this work, we study the neural network pruning in the data-free context, aiming to yield lightweight models that are not only accurate in prediction but also robust against undesired inputs in open-world deployments. Considering the absence of the fine-tuning and retraining that can fix the mis-pruned units, we replace the traditional aggressive one-shot strategy with a conservative one that treats the pruning as a progressive process. We propose a pruning method based on stochastic optimization that uses robustness-related metrics to guide the pruning process. Our method is implemented as a Python package named PAODING and evaluated with a series of experiments on diverse neural network models. The experimental results show that it significantly outperforms existing one-shot data-free pruning approaches in terms of robustness preservation and accuracy.Preprint. Under review.

show abstract