CHEX: CHannel EXploration for CNN Model Compression

Hou, Zejiang; Qin, Minghai; Sun, Fei; Ma, Xiaolong; Yuan, Kun; Xu, Yi; Chen, Yen-Kuang; Jin, Rong; Xie, Yuan; Kung, Sun‐Yuan

doi:10.1109/cvpr52688.2022.01197

Cited by 55 publications

(31 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For spatiotemporal learning, BEVT [75] and VideoMAE [70,25] can be seen as extensions of BeiT and MAE, respectively. Recent works also indicate that CLIP features provide good guidance for mask modeling [78,33,60,59,84], but all of them actually perform worse than CLIP itself with elaborate fine-tuning [21]. In contrast, we demonstrate that in the video domain, our model with CLIP supervision clearly outperforms the teacher.…”

Section: Related Workmentioning

confidence: 56%

“…However, the aggressive random masking may only retain the background tokens, which contain insignificant information and hinder the teacher's knowledge transfer. To enhance target effectiveness, we apply the semantic masking [33] frame by frame, where the tokens with important clues are maintained at higher probabilities. Specifically, given the class token z cls ∈R 1×C and the spatial tokens Z∈R L×C in the tth frame of CLIP-ViT (L=H×W is the token number and C is the token dimension), we calculate the attention score in the last self-attention [23] layer:…”

Section: Unmasked Teachermentioning

confidence: 99%

See 1 more Smart Citation

Unmasked Teacher: Towards Training-Efficient Video Foundation Models

Li¹,

Wang²,

Li³

et al. 2023

Preprint

View full text Add to dashboard Cite

Video Foundation Models (VFMs) have received limited exploration due to high computational costs and data scarcity. Previous VFMs rely on Image Foundation Models (IFMs), which face challenges in transferring to the video domain. Although VideoMAE has trained a robust ViT from limited data, its low-level reconstruction poses convergence difficulties and conflicts with high-level cross-modal alignment. This paper proposes a training-efficient method for temporal-sensitive VFMs that integrates the benefits of existing methods. To increase data efficiency, we mask out most of the low-semantics video tokens, but selectively align the unmasked tokens with IFM, which serves as the UnMasked Teacher (UMT). By providing semantic guidance, our method enables faster convergence and multimodal friendliness. With a progressive pre-training framework, our model can handle various tasks including scenerelated, temporal-related, and complex video-language understanding. Using only public sources for pre-training in 6 days on 32 A100 GPUs, our scratch-built ViT-L/16 achieves state-of-the-art performances on various video tasks. The code and models will be released at https://github. com/OpenGVLab/unmasked_teacher.

show abstract

Section: Related Workmentioning

confidence: 56%

Section: Unmasked Teachermentioning

confidence: 99%

Unmasked Teacher: Towards Training-Efficient Video Foundation Models

Li¹,

Wang²,

Li³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Compared with the stateof-the-art automatic low-rank compression method ALDS, HALOC enjoys 1.4% higher top-1 accuracy with much lower computational costs. Compared with the state-of-theart pruning work CHEX (Hou et al 2022), HALOC also achieves 0.54% accuracy increase with much fewer FLOPs. Besides, our approach also shows impressive performance for compressing MobileNetV2, a task that is conventionally challenging for low-rank compression approach.…”

Section: Imagenet Resultsmentioning

confidence: 99%

HALOC: Hardware-Aware Automatic Low-Rank Compression for Compact Neural Networks

Xiao¹,

Zhang²,

Gong³

et al. 2023

Preprint

View full text Add to dashboard Cite

Low-rank compression is an important model compression strategy for obtaining compact neural network models. In general, because the rank values directly determine the model complexity and model accuracy, proper selection of layerwise rank is very critical and desired. To date, though many low-rank compression approaches, either selecting the ranks in a manual or automatic way, have been proposed, they suffer from costly manual trials or unsatisfied compression performance. In addition, all of the existing works are not designed in a hardware-aware way, limiting the practical performance of the compressed models on real-world hardware platforms. To address these challenges, in this paper we propose HALOC, a hardware-aware automatic low-rank compression framework. By interpreting automatic rank selection from an architecture search perspective, we develop an end-to-end solution to determine the suitable layer-wise ranks in a differentiable and hardware-aware way. We further propose design principles and mitigation strategy to efficiently explore the rank space and reduce the potential interference problem. Experimental results on different datasets and hardware platforms demonstrate the effectiveness of our proposed approach. On CIFAR-10 dataset, HALOC enables 0.07% and 0.38% accuracy increase over the uncompressed ResNet-20 and VGG-16 models with 72.20% and 86.44% fewer FLOPs, respectively. On ImageNet dataset, HALOC achieves 0.9% higher top-1 accuracy than the original ResNet-18 model with 66.16% fewer FLOPs. HALOC also shows 0.66% higher top-1 accuracy increase than the state-of-the-art automatic low-rank compression solution with fewer computational and memory costs. In addition, HALOC demonstrates the practical speedups on different hardware platforms, verified by the measurement results on desktop GPU, embedded GPU and ASIC accelerator.

show abstract

“…Shen et al [ 42 ] pruned channels globally based on magnitude and gradient criteria. Unlike pruning-only methods, Hou et al [ 43 ] proposed a pruning-and-regrowing method to avoid removing important channels. These methods can compress a network while ensuring high performance.…”

Section: Related Workmentioning

confidence: 99%

Lossless Reconstruction of Convolutional Neural Network for Channel-Based Network Pruning

Lee

Hwang

2023

Sensors

View full text Add to dashboard Cite

Network pruning reduces the number of parameters and computational costs of convolutional neural networks while maintaining high performance. Although existing pruning methods have achieved excellent results, they do not consider reconstruction after pruning in order to apply the network to actual devices. This study proposes a reconstruction process for channel-based network pruning. For lossless reconstruction, we focus on three components of the network: the residual block, skip connection, and convolution layer. Union operation and index alignment are applied to the residual block and skip connection, respectively. Furthermore, we reconstruct a compressed convolution layer by considering batch normalization. We apply our method to existing channel-based pruning methods for downstream tasks such as image classification, object detection, and semantic segmentation. Experimental results show that compressing a large model has a 1.93% higher accuracy in image classification, 2.2 higher mean Intersection over Union (mIoU) in semantic segmentation, and 0.054 higher mean Average Precision (mAP) in object detection than well-designed small models. Moreover, we demonstrate that our method can reduce the actual latency by 8.15× and 5.29× on Raspberry Pi and Jetson Nano, respectively.

show abstract

CHEX: CHannel EXploration for CNN Model Compression

Cited by 55 publications

References 40 publications

Unmasked Teacher: Towards Training-Efficient Video Foundation Models

Unmasked Teacher: Towards Training-Efficient Video Foundation Models

HALOC: Hardware-Aware Automatic Low-Rank Compression for Compact Neural Networks

Lossless Reconstruction of Convolutional Neural Network for Channel-Based Network Pruning

Contact Info

Product

Resources

About