Deep Pyramidal Residual Networks

Han, Dongil; Kim, Jiwhan; Kim, Junmo

doi:10.1109/cvpr.2017.668

Cited by 564 publications

(404 citation statements)

References 16 publications

Supporting

Mentioning

401

Contrasting

Order By: Relevance

“…We evaluate self distillation on five convolutional neural networks (ResNet [14], WideResNet [41], Pyramid ResNet [11], ResNeXt [38], VGG [34]) and two datasets (CIFAR100 [21], ImageNet [6]). Learning rate decay, l 2 regularizer and simple data argumentation are used during the training process.…”

Section: Methodsmentioning

confidence: 99%

Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation

Zhang

Song²,

Gao³

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

698

375

View full text Add to dashboard Cite

Convolutional neural networks have been widely deployed in various application scenarios. In order to extend the applications' boundaries to some accuracy-crucial domains, researchers have been investigating approaches to boost accuracy through either deeper or wider network structures, which brings with them the exponential increment of the computational and storage cost, delaying the responding time.In this paper, we propose a general training framework named self distillation, which notably enhances the performance (accuracy) of convolutional neural networks through shrinking the size of the network rather than aggrandizing it. Different from traditional knowledge distillation -a knowledge transformation methodology among networks, which forces student neural networks to approximate the softmax layer outputs of pre-trained teacher neural networks, the proposed self distillation framework distills knowledge within network itself. The networks are firstly divided into several sections. Then the knowledge in the deeper portion of the networks is squeezed into the shallow ones. Experiments further prove the generalization of the proposed self distillation framework: enhancement of accuracy at average level is 2.65%, varying from 0.61% in ResNeXt as minimum to 4.07% in VGG19 as maximum. In addition, it can also provide flexibility of depth-wise scalable inference on resource-limited edge devices. Our codes will be released on github soon.

show abstract

Section: Methodsmentioning

confidence: 99%

Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation

Zhang

Song²,

Gao³

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

698

375

View full text Add to dashboard Cite

show abstract

“…Note that, due to the separability of the basis functions, 2D STFT can be efficiently computed using simple 1D convolutions for the rows and the columns, successively. Using vector notation, we can rewrite Equation 2 as shown in Equation 3.…”

Section: Methodsmentioning

confidence: 99%

Depthwise-STFT Based Separable Convolutional Neural Networks

Kumawat

Raman

2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

In this paper, we propose a new convolutional layer called Depthwise-STFT Separable layer that can serve as an alternative to the standard depthwise separable convolutional layer. The construction of the proposed layer is inspired by the fact that the Fourier coefficients can accurately represent important features such as edges in an image. It utilizes the Fourier coefficients computed (channelwise) in the 2D local neighborhood (e.g., 3 × 3) of each position of the input map to obtain the feature maps. The Fourier coefficients are computed using 2D Short Term Fourier Transform (STFT) at multiple fixed low frequency points in the 2D local neighborhood at each position. These feature maps at different frequency points are then linearly combined using trainable pointwise (1 × 1) convolutions. We show that the proposed layer outperforms the standard depthwise separable layer based models on the CIFAR-10 and CIFAR-100 image classification datasets with reduced space-time complexity.

show abstract

“…III-B2, and denote it as 'MW', while the original CNN is denoted as 'Base'. Table VIII shows the detailed results of accuracy with the competing methods PreResNet [10], All-CNN [73], WideRes-Net [53], PyramidNet [50], DenseNet [52] on CIFAR-10, CIFAR-100, SVHN, MNIST and ImageNet32. Table IX and Table X show Top-1 and Top-5 error of ResNet [9] on imagenet64 and Place365.…”

Section: Extend To Object Classificationmentioning

confidence: 99%

Multi-Level Wavelet Convolutional Neural Networks

et al. 2019

View full text Add to dashboard Cite

In computer vision, convolutional networks (CNNs) often adopts pooling to enlarge receptive field which has the advantage of low computational complexity. However, pooling can cause information loss and thus is detrimental to further operations such as features extraction and analysis. Recently, dilated filter has been proposed to trade off between receptive field size and efficiency. But the accompanying gridding effect can cause a sparse sampling of input images with checkerboard patterns. To address this problem, in this paper, we propose a novel multi-level wavelet CNN (MWCNN) model to achieve better trade-off between receptive field size and computational efficiency. The core idea is to embed wavelet transform into CNN architecture to reduce the resolution of feature maps while at the same time, increasing receptive field. Specifically, MWCNN for image restoration is based on U-Net architecture, and inverse wavelet transform (IWT) is deployed to reconstruct the high resolution (HR) feature maps. The proposed MWCNN can also be viewed as an improvement of dilated filter and a generalization of average pooling, and can be applied to not only image restoration tasks, but also any CNNs requiring a pooling operation. The experimental results demonstrate effectiveness of the proposed MWCNN for tasks such as image denoising, single image super-resolution, JPEG image artifacts removal and object classification. The code and pre-trained models will be given at https://github.com/lpj-github-io/MWCNNv2. Index Terms-Convolutional networks, receptive field size, efficiency, multi-level wavelet.

show abstract

Deep Pyramidal Residual Networks

Abstract: Deep convolutional neural networks (DCNNs) have shown remarkable performance in image classification tasks

Cited by 564 publications

References 16 publications

Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation

Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation

Depthwise-STFT Based Separable Convolutional Neural Networks

Multi-Level Wavelet Convolutional Neural Networks

Contact Info

Product

Resources

About