Dynamic slicing for deep neural networks

Zhang, Ziqi; Li, Yuanchun; Guo, Yao; Chen, Xiangqun; Liu, Yunxin

doi:10.1145/3368089.3409676

Cited by 31 publications

(8 citation statements)

References 81 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Model pruning is a promising solution to accelerate CNNs by removing unimportant weights (Han et al 2015;Wen et al 2016;Lebedev and Lempitsky 2016;Liu, Guo, and Chen 2019;Zhang et al 2020;Liu et al 2020). Here we focus on filter pruning due to its applicability to any CNN architectures without requiring special software/hardware accelerators.…”

Section: Model Pruningmentioning

confidence: 99%

TransTailor: Pruning the Pre-trained Model for Improved Transfer Learning

Liu

Cai

Guo

et al. 2021

AAAI

Self Cite

View full text Add to dashboard Cite

The increasing of pre-trained models has significantly facilitated the performance on limited data tasks with transfer learning. However, progress on transfer learning mainly focuses on optimizing the weights of pre-trained models, which ignores the structure mismatch between the model and the target task. This paper aims to improve the transfer performance from another angle - in addition to tuning the weights, we tune the structure of pre-trained models, in order to better match the target task. To this end, we propose TransTailor, targeting at pruning the pre-trained model for improved transfer learning. Different from traditional pruning pipelines, we prune and fine-tune the pre-trained model according to the target-aware weight importance, generating an optimal sub-model tailored for a specific target task. In this way, we transfer a more suitable sub-structure that can be applied during fine-tuning to benefit the final performance. Extensive experiments on multiple pre-trained models and datasets demonstrate that TransTailor outperforms the traditional pruning methods and achieves competitive or even better performance than other state-of-the-art transfer learning methods while using a smaller model. Notably, on the Stanford Dogs dataset, TransTailor can achieve 2.7% accuracy improvement over other transfer methods with 20% fewer FLOPs.

show abstract

Section: Model Pruningmentioning

confidence: 99%

TransTailor: Pruning the Pre-trained Model for Improved Transfer Learning

Liu

Cai

Guo

et al. 2021

AAAI

Self Cite

View full text Add to dashboard Cite

show abstract

“…To protect the IP right of a model, one way is to avoid model exposure by encrypting it [21], putting it or part of it into enclaves [70,79], etc. Another way is to design mechanisms to enable model IP violation detection.…”

Section: Model Intellectual Property Protectionmentioning

confidence: 99%

ModelDiff: testing-based DNN similarity comparison for model reuse detection

Zhang

Liu

et al. 2021

Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis

Self Cite

View full text Add to dashboard Cite

The knowledge of a deep learning model may be transferred to a student model, leading to intellectual property infringement or vulnerability propagation. Detecting such knowledge reuse is nontrivial because the suspect models may not be white-box accessible and/or may serve different tasks. In this paper, we propose Mod-elDiff, a testing-based approach to deep learning model similarity comparison. Instead of directly comparing the weights, activations, or outputs of two models, we compare their behavioral patterns on the same set of test inputs. Specifically, the behavioral pattern of a model is represented as a decision distance vector (DDV), in which each element is the distance between the model's reactions to a pair of inputs. The knowledge similarity between two models is measured with the cosine similarity between their DDVs. To evaluate ModelDiff, we created a benchmark that contains 144 pairs of models that cover most popular model reuse methods, including transfer learning, model compression, and model stealing. Our method achieved 91.7% correctness on the benchmark, which demonstrates the effectiveness of using ModelDiff for model reuse detection. A study on mobile deep learning apps has shown the feasibility of ModelDiff on real-world models. CCS CONCEPTS• Security and privacy → Software and application security; Digital rights management; • Software and its engineering → Software post-development issues.

show abstract

“…Our pruning method is designed to pursue robustness preservation, given that the model may be exposed to unexpected or even adversarial inputs [22,23,24,25,26] after being deployed in a real-world application scenario.…”

Section: Introductionmentioning

confidence: 99%

Supervised Robustness-preserving Data-free Neural Network Pruning

Meng¹,

Bai²,

Teo³

et al. 2022

Preprint

View full text Add to dashboard Cite

When deploying pre-trained neural network models in real-world applications, model consumers often encounter resource-constraint platforms such as mobile and smart devices. They typically use the pruning technique to reduce the size and complexity of the model, generating a lighter one with less resource consumption. Nonetheless, most existing pruning methods are proposed with a premise that the model after being pruned has a chance to be fine-tuned or even retrained based on the original training data. This may be unrealistic in practice, as the data controllers are often reluctant to provide their model consumers with the original data. In this work, we study the neural network pruning in the data-free context, aiming to yield lightweight models that are not only accurate in prediction but also robust against undesired inputs in open-world deployments. Considering the absence of the fine-tuning and retraining that can fix the mis-pruned units, we replace the traditional aggressive one-shot strategy with a conservative one that treats the pruning as a progressive process. We propose a pruning method based on stochastic optimization that uses robustness-related metrics to guide the pruning process. Our method is implemented as a Python package named PAODING and evaluated with a series of experiments on diverse neural network models. The experimental results show that it significantly outperforms existing one-shot data-free pruning approaches in terms of robustness preservation and accuracy.Preprint. Under review.

show abstract

Dynamic slicing for deep neural networks

Cited by 31 publications

References 81 publications

TransTailor: Pruning the Pre-trained Model for Improved Transfer Learning

TransTailor: Pruning the Pre-trained Model for Improved Transfer Learning

ModelDiff: testing-based DNN similarity comparison for model reuse detection

Supervised Robustness-preserving Data-free Neural Network Pruning

Contact Info

Product

Resources

About