SGAD: Soft-Guided Adaptively-Dropped Neural Network

Abstract

Deep neural networks (DNNs) have been proven to have many redundancies. Hence, many efforts have been made to compress DNNs. However, the existing model compression methods treat all the input samples equally while ignoring the fact that the difficulties of various input samples being correctly classified are different. To address this problem, DNNs with adaptive dropping mechanism are well explored in this work. To inform the DNNs how difficult the input samples can be classified, a guideline that contains the information of input samples is introduced to improve the performance. Based on the developed guideline and adaptive dropping mechanism, an innovative soft-guided adaptively-dropped (SGAD) neural network is proposed in this paper. Compared with the 32 layers residual neural networks, the presented SGAD can reduce the FLOPs by 77% with less than 1% drop in accuracy on CIFAR-10.

Design Light-weight 3D Convolutional Networks for Video Recognition Temporal Residual, Fully Separable Block, and Fast Algorithm

“…We suppose that the slow path mainly play a role of capturing features from still images and providing the object instances infor-mation to support the final action recognition, but the slow path dominates the overall memory footprint of the network (≈ 98%). According to the general perspective of many works that focus on model compression of 2D Con-vNets [10,26], we believe that there will be considerable redundancy in the slow path, so a channel bottleneck structure are introduced, which is composed of a depth-wise 3 × 3 kernel and a point-wise 1 × 1 one, to replace the 1×3×3 kernel for further reduction of the model size. This variant is denoted as SlowDepth in Fig.…”