Deep Networks with Stochastic Depth

Huang, Gao; Sun, Yu; Liu, Zhuang; Sedra, Daniel; Weinberger, Kilian Q.

doi:10.1007/978-3-319-46493-0_39

Cited by 1,526 publications

(1,214 citation statements)

References 23 publications

Supporting

Mentioning

1,203

Contrasting

Unclassified

Order By: Relevance

“…The comparison is provided in Table 2. Our method has some accuracy degradation in comparison to state-of-the-art supervised publication [45], which has increased the considerable depth of residual networks even beyond 1200 layers. The layers of 1200 are an astronomical figure.…”

Section: Methodsmentioning

confidence: 96%

“…Accuracy L2 sparse filtering [40] 63.89% 3-way factored RBM (3 layers) [41] 65.30% Mc RBM (3 layers) [42] 71.00% Tiled CNN [43] 73.10% Stochastic pooling ConvNet [44] 84.87% Deep networks with stochastic depth [45] 95.09% Our method 74.18% different feature numbers and sparsity parameter values.…”

Section: Methodsmentioning

confidence: 99%

“…To prove that our method is of low computational cost than some state-of-the-art methods, we focus on two representative baselines--Stochastic pooling ConvNet [44] and Deep networks with stochastic depth [45]. These two models compared are the current state-of-the-art CNNs, which outperform our method on classification accuracy.…”

Section: Analysis Of Computational Complexitymentioning

confidence: 99%

See 2 more Smart Citations

Image Classification Based on Convolutional Denoising Sparse Autoencoder

Chen

Liu

Zeng

et al. 2017

Mathematical Problems in Engineering

View full text Add to dashboard Cite

Image classification aims to group images into corresponding semantic categories. Due to the difficulties of interclass similarity and intraclass variability, it is a challenging issue in computer vision. In this paper, an unsupervised feature learning approach called convolutional denoising sparse autoencoder (CDSAE) is proposed based on the theory of visual attention mechanism and deep learning methods. Firstly, saliency detection method is utilized to get training samples for unsupervised feature learning. Next, these samples are sent to the denoising sparse autoencoder (DSAE), followed by convolutional layer and local contrast normalization layer. Generally, prior in a specific task is helpful for the task solution. Therefore, a new pooling strategy-spatial pyramid pooling (SPP) fused with center-bias prior-is introduced into our approach. Experimental results on the common two image datasets (STL-10 and CIFAR-10) demonstrate that our approach is effective in image classification. They also demonstrate that none of these three components: local contrast normalization, SPP fused with center-prior, and 2 vector normalization can be excluded from our proposed approach. They jointly improve image representation and classification performance.

show abstract

Section: Methodsmentioning

confidence: 96%

Section: Methodsmentioning

confidence: 99%

Section: Analysis Of Computational Complexitymentioning

confidence: 99%

See 1 more Smart Citation

Image Classification Based on Convolutional Denoising Sparse Autoencoder

Chen

Liu

Zeng

et al. 2017

Mathematical Problems in Engineering

View full text Add to dashboard Cite

show abstract

“…The ensemble method is also an effective technique for reducing generalization errors by combining multiple models. Huang et al [31] recently developed a training algorithm that drops a random subset of layers into a deep residual network [20] and achieves good performance in reducing overfitting. Bastien et al [32] developed a powerful generator of stochastic variations for handwritten character images.…”

Section: Related Studiesmentioning

confidence: 99%

DropRegion training of inception font network for high-performance Chinese font recognition

Huang

Zhong

Jin

et al. 2018

Pattern Recognition

View full text Add to dashboard Cite

Chinese font recognition (CFR) has gained significant attention in recent years. However, due to the sparsity of labeled font samples and the structural complexity of Chinese characters, CFR is still a challenging task. In this paper, a DropRegion method is proposed to generate a large number of stochastic variant font samples whose local regions are selectively disrupted and an inception font network (IFN) with two additional convolutional neural network (CNN) structure elements, i.e., a cascaded cross-channel parametric pooling (CCCP) and global average pooling, is designed. Because the distribution of strokes in a font image is non-stationary, an elastic meshing technique that adaptively constructs a set of local regions with equalized information is developed. Thus, DropRegion is seamlessly embedded in the IFN, which enables end-to-end training; the proposed DropRegion -IFN can be used for high performance CFR. Experimental results have confirmed the effectiveness of our new approach for CFR.

show abstract

“…The ResNet was also shown to alleviate optimization difficulty in training very deep networks. In [34], identity shortcut connections were used for bypassing a subset of layers to facilitate training very deep networks.…”

Section: Related Workmentioning

confidence: 99%

Training Very Deep Networks via Residual Learning with Stochastic Input Shortcut Connections

Oyedotun

Shabayek

Aouada

et al. 2017

Neural Information Processing

View full text Add to dashboard Cite

Abstract. Many works have posited the benefit of depth in deep networks. However, one of the problems encountered in the training of very deep networks is feature reuse; that is, features are 'diluted' as they are forward propagated through the model. Hence, later network layers receive less informative signals about the input data, consequently making training less effective. In this work, we address the problem of feature reuse by taking inspiration from an earlier work which employed residual learning for alleviating the problem of feature reuse. We propose a modification of residual learning for training very deep networks to realize improved generalization performance; for this, we allow stochastic shortcut connections of identity mappings from the input to hidden layers. We perform extensive experiments using the USPS and MNIST datasets. On the USPS dataset, we achieve an error rate of 2.69% without employing any form of data augmentation (or manipulation). On the MNIST dataset, we reach a comparable state-of-the-art error rate of 0.52%. Particularly, these results are achieved without employing any explicit regularization technique.

show abstract

Deep Networks with Stochastic Depth

Cited by 1,526 publications

References 23 publications

Image Classification Based on Convolutional Denoising Sparse Autoencoder

Image Classification Based on Convolutional Denoising Sparse Autoencoder

DropRegion training of inception font network for high-performance Chinese font recognition

Training Very Deep Networks via Residual Learning with Stochastic Input Shortcut Connections

Contact Info

Product

Resources

About