Identity Mappings in Deep Residual Networks

Abstract

Deep residual networks [1] have emerged as a family of extremely deep architectures showing compelling accuracy and nice convergence behaviors. In this paper, we analyze the propagation formulations behind the residual building blocks, which suggest that the forward and backward signals can be directly propagated from one block to any other block, when using identity mappings as the skip connections and after-addition activation. A series of ablation experiments support the importance of these identity mappings. This motivates us to propose a new residual unit, which makes training easier and improves generalization. We report improved results using a 1001-layer ResNet on CIFAR-10 (4.62% error) and CIFAR-100, and a 200-layer ResNet on ImageNet. Code is available at: https://github.com/KaimingHe/ resnet-1k-layers.

Energy-efficient Amortized Inference with Cascaded Deep Classifiers

“…Conditional computation [Bengio et al, 2015] and adaptive computation [Graves, 2016] propose to adjust the amount of computational cost by using a policy to select data. Many of these static and dynamic techniques are used in standard deep architectures such as ResNet [He et al, 2016a] and Inception [Szegedy et al, 2017], usually with a loss of accuracy. Different from these static and dynamic techniques, our method explicitly formulates the testtime efficiency as an amortized constrained sequential decision problem such that the expected computational cost, in terms of FLOPs cost, can be greatly reduced with even improved accuracy by adaptively assigning training examples with various difficulty to their best classifiers.…”

“…Deep residual network [He et al, 2016a] has been widely used in image classification field since it was proposed. We choose ResNet as our model's baseline because we can easily build a sequence of networks from shallow to deep by adjusting the number of units in each block.…”

“…Since the first success of deep convolutional neural network in the ImageNet challenge, more complex architectures [Simonyan and Zisserman, 2014;He et al, 2016a;Szegedy et al, 2017] have been proposed to further improve performance, but often at the cost of more expensive computation. However, in many real-world scenarios, such as vision-based robotics and mobile vision applications, we encounter a significant constraint of energy or computational cost for real-time inference.…”

Artificial Neural Networks and Machine Learning – ICANN 2017

“…One of such architectures is a novel ultra-deep residual learning network (ResNet) [4]. This architecture can be implemented by adding so called 'shortcut connections' [5] which skip one or more layers. They perform a mapping so that their outputs are added to the outputs of the stacked layes.…”

Neural Networks and Spelling Features for Native Language Identification

“…For one, spelling errors are quite prevalent in learners' written production (Kochmar, 2011). Additionally, spelling errors have been shown to be influenced by phonological L1 transfer (Grigonytė and Hammarberg, 2014 (He et al, 2015(He et al, , 2016. Such skip connections facilitate error propagation to earlier layers in the network, which allows for building deeper networks.…”