VR-DANN: Real-Time Video Recognition via Decoder-Assisted Neural Network Acceleration

Song, Zhuoran; Wu, Feiyang; Liu, Xueyuan; Jin, Ke; Jing, Naifeng; Liang, Xiaoyao

doi:10.1109/micro50266.2020.00063

Cited by 16 publications

(1 citation statement)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, Transformers have demonstrated great successes in computer vision, such as image classification [10,16,20,21,23], object detection [3,4,24,27], semantic segmentation [1,19], and action recognition [7,13,15]. Thanks to the self-attention based architectures, Vision Transformers (ViT) [6] outperforms the classical Convolutional Neural Networks (CNN) [17] which achieves the state-of-theart results.…”

Section: Introductionmentioning

confidence: 99%

CP-ViT: Cascade Vision Transformer Pruning via Progressive Sparsity Prediction

Song¹,

Xu²,

He³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Vision transformer (ViT) has achieved competitive accuracy on a variety of computer vision applications, but its computational cost impedes the deployment on resourcelimited mobile devices. We explore the sparsity in ViT and observe that informative patches and heads are sufficient for accurate image recognition. In this paper, we propose a cascade pruning framework named CP-ViT by predicting sparsity in ViT models progressively and dynamically to reduce computational redundancy while minimizing the accuracy loss. Specifically, we define the cumulative score to reserve the informative patches and heads across the ViT model for better accuracy. We also propose the dynamic pruning ratio adjustment technique based on layer-aware attention range. CP-ViT has great general applicability for practical deployment, which can be applied to a wide range of ViT models and can achieve superior accuracy with or without fine-tuning. Extensive experiments on ImageNet, CIFAR-10, and CIFAR-100 with various pre-trained models have demonstrated the effectiveness and efficiency of CP-ViT. By progressively pruning 50% patches, our CP-ViT method reduces over 40% FLOPs while maintaining accuracy loss within 1%.

show abstract