Residual Connections Encourage Iterative Inference

Jastrzębski, Stanisław; Arpit, Devansh; Ballas, Nicolas; Verma, Vikas; Che, Tong; Bengio, Yoshua

doi:10.48550/arxiv.1710.04773

Cited by 21 publications

(35 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Such phenomenon shows that the first several blocks in a C-block have relatively higher LCMC than the others. The discovery gained here is consistent with the reports from previous works Veit et al [2016]; Jastrzebski et al [2017].…”

Section: Comparisons and Discussionsupporting

confidence: 94%

SGAD: Soft-Guided Adaptively-Dropped Neural Network

Wang,

Sun,

Lin

et al. 2018

Preprint

View full text Add to dashboard Cite

Deep neural networks (DNNs) have been proven to have many redundancies. Hence, many efforts have been made to compress DNNs. However, the existing model compression methods treat all the input samples equally while ignoring the fact that the difficulties of various input samples being correctly classified are different. To address this problem, DNNs with adaptive dropping mechanism are well explored in this work. To inform the DNNs how difficult the input samples can be classified, a guideline that contains the information of input samples is introduced to improve the performance. Based on the developed guideline and adaptive dropping mechanism, an innovative soft-guided adaptively-dropped (SGAD) neural network is proposed in this paper. Compared with the 32 layers residual neural networks, the presented SGAD can reduce the FLOPs by 77% with less than 1% drop in accuracy on CIFAR-10.

show abstract

Section: Comparisons and Discussionsupporting

confidence: 94%

SGAD: Soft-Guided Adaptively-Dropped Neural Network

Wang,

Sun,

Lin

et al. 2018

Preprint

View full text Add to dashboard Cite

show abstract

“…Relatedly, residual blocks, a popular architectural pattern in feedforward models, might provide an inductive bias similar to recurrence (Liao & Poggio, 2016), enabling an equivalent form of iterative processing, which could explain their increased efficacy in categorization (compared to vanilla feedforward models), especially for ambiguous images (Jastrzębski et al, 2017).…”

Section: Discussionmentioning

confidence: 99%

Category-orthogonal object features guide information processing in recurrent neural networks trained for object categorization

Thorat¹,

Aldegheri²,

Kietzmann³

2021

Preprint

View full text Add to dashboard Cite

Recurrent neural networks (RNNs) have been shown to perform better than feedforward architectures in visual object categorization tasks, especially in challenging conditions such as cluttered images. However, little is known about the exact computational role of recurrent information flow in these conditions. Here we test RNNs trained for object categorization on the hypothesis that recurrence iteratively aids object categorization via the communication of category-orthogonal auxiliary variables (the location, orientation, and scale of the object). Using diagnostic linear readouts, we find that: (a) information about auxiliary variables increases across time in all network layers, (b) this information is indeed present in the recurrent information flow, and (c) its manipulation significantly affects task performance. These observations confirm the hypothesis that category-orthogonal auxiliary variable information is conveyed through recurrent connectivity and is used to optimize category inference in cluttered environments. † * Equal contribution. † Code and data: https://github.com/KietzmannLab/svrhm21_RNN_explain 3rd Workshop on Shared Visual Representations in Human and Machine Intelligence (SVRHM 2021) of the Neural Information Processing Systems (NeurIPS) conference, Virtual.

show abstract

“…Since this first block is different from the rest, it cannot be folded. These projection blocks could be eliminated, as explored in [21] and [15], but following the intuition that these dimensional transformations correspond to compositional changes in the level of representation [5], this paper keeps them. The rest of the blocks are folded into a single recurrent block, which is iterated a number of times equal to the number of blocks folded (see Figure 1c).…”

Section: Proposed Method: Hidden-fold Networkmentioning

confidence: 99%

“…However, folded ResNets suffer from overfitting and exploding layer activations. Reference [15] suggested using unshared batch normalization (UBN) to alleviate this problem. UBN consists of not sharing batch normalization's learnable parameters (β and γ) between folded blocks, having a different set for each iteration instead, as exemplified in Figure 3.…”

Section: Proposed Method: Hidden-fold Networkmentioning

confidence: 99%

“…A different method of compressing networks is found in the hypothesis that ResNets may be approximating unrolled shallow recursive neural networks (RNNs), and that the gains from additional layers correspond to additional recursive iterations [5,21]. This theory is supported by the authors of [15], who found that ResNet's deeper residual blocks are learning to perform iterative refinement of features. Moreover, the work in [21] demonstrated that folding ResNet into a 4-layer RNN only had a moderate impact on accuracy.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Hidden-Fold Networks: Random Recurrent Residuals Using Sparse Supermasks

Garcia-Arias¹,

Hashimoto²,

Motomura³

et al. 2021

Preprint

View full text Add to dashboard Cite

Deep neural networks (DNNs) are so over-parametrized that recent research has found them to already contain a subnetwork with high accuracy at their randomly initialized state. Finding these subnetworks is a viable alternative training method to weight learning. In parallel, another line of work has hypothesized that deep residual networks (ResNets) are trying to approximate the behaviour of shallow recurrent neural networks (RNNs) and has proposed a way for compressing them into recurrent models. This paper proposes blending these lines of research into a highly compressed yet accurate model: Hidden-Fold Networks (HFNs). By first folding ResNet into a recurrent structure and then searching for an accurate subnetwork hidden within the randomly initialized model, a high-performing yet tiny HFN is obtained without ever updating the weights. As a result, HFN achieves equivalent performance to ResNet50 on CIFAR100 while occupying 38.5x less memory, and similar performance to ResNet34 on ImageNet with a memory size 26.8x smaller. The HFN will become even more attractive by minimizing data transfers while staying accurate when it runs on highly-quantized and randomlyweighted DNN inference accelerators.

show abstract

Residual Connections Encourage Iterative Inference

Cited by 21 publications

References 8 publications

SGAD: Soft-Guided Adaptively-Dropped Neural Network

SGAD: Soft-Guided Adaptively-Dropped Neural Network

Category-orthogonal object features guide information processing in recurrent neural networks trained for object categorization

Hidden-Fold Networks: Random Recurrent Residuals Using Sparse Supermasks

Contact Info

Product

Resources

About