Building a Regular Decision Boundary with Deep Networks

Oyallon, Edouard

doi:10.1109/cvpr.2017.204

Cited by 12 publications

(10 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The i-RevNet is constructed through a cascade of homeomorphic layers which can be fully inverted, with an explicit formula, up to the last hidden layer and therefore no information is discarded. The phenomenon of progressive separation and contraction in non-invertible networks is also observed in several other works [15,21].…”

supporting

confidence: 72%

Fixed-point algorithms for inverse of residual rectifier neural networks

Wang¹,

An²,

Liu³

et al. 2021

MFC

View full text Add to dashboard Cite

A deep neural network with invertible hidden layers has a nice property of preserving all the information in the feature learning stage. In this paper, we analyse the hidden layers of residual rectifier neural networks, and investigate conditions for invertibility under which the hidden layers are invertible. A new fixed-point algorithm is developed to invert the hidden layers of residual networks. The proposed inverse algorithms are capable of inverting some residual networks which cannot be inverted by existing inverting algorithms. Furthermore, a special residual rectifier network is designed and trained on MNIST so that it can achieve comparable performance with the state-of-art performance while its hidden layers are invertible. 2020 Mathematics Subject Classification. Primary: 68T07. Key words and phrases. Invertible neural networks, fixed-point algorithms, invertible residual networks, invertible rectifier neural networks, inverse algorithm of neural networks.

show abstract

supporting

confidence: 72%

Fixed-point algorithms for inverse of residual rectifier neural networks

Wang¹,

An²,

Liu³

et al. 2021

MFC

View full text Add to dashboard Cite

show abstract

“…3.1 for this case shows that the layerwise k = 1 procedure will try to progressively improve the linear separability. Progressive linear separation has been empirically studied in end-to-end CNNs (Zeiler & Fergus, 2014;Oyallon, 2017) as an indirect consequence, while the k = 1 training permits us to study this basic principle more directly as the layer objective. Concurrent work (Elad et al, 2019) follows the same argument to use a layerwise training procedure to evaluate mutual information more directly.…”

Section: Auxiliary Problems and The Properties They Inducementioning

confidence: 99%

“…First, the use of a global objective means that the final functional behavior of individual intermediate layers of a deep network is only indirectly specified: it is unclear how the layers work together to achieve high-accuracy predictions. Several authors have suggested and shown empirically that CNNs learn to implement mechanisms that progressively induce invariance to complex, but irrelevant variability (Mallat, 2016;Yosinski et al, 2015) while increasing linear separability (Zeiler & Fergus, 2014;Oyallon, 2017;Jacobsen et al, 2018) of the data. Progressive linear separability has been shown empirically but it is unclear whether this is merely the consequence of other strategies implemented by CNNs, or if it is a sufficient condition for the high performance of these networks.…”

Section: Introductionmentioning

confidence: 99%

Greedy Layerwise Learning Can Scale to ImageNet

Belilovsky¹,

Eickenberg²,

Oyallon³

2018

Preprint

Self Cite

View full text Add to dashboard Cite

Shallow supervised 1-hidden layer neural networks have a number of favorable properties that make them easier to interpret, analyze, and optimize than their deep counterparts, but lack their representational power. Here we use 1-hidden layer learning problems to sequentially build deep networks layer by layer, which can inherit properties from shallow networks. Contrary to previous approaches using shallow networks, we focus on problems where deep learning is reported as critical for success. We thus study CNNs on image classification tasks using the large-scale ImageNet dataset and the CIFAR-10 dataset. Using a simple set of ideas for architecture and training we find that solving sequential 1-hidden-layer auxiliary problems lead to a CNN that exceeds AlexNet performance on ImageNet. Extending this training methodology to construct individual layers by solving 2-and-3-hidden layer auxiliary problems, we obtain an 11-layer network that exceeds several members of the VGG model family on ImageNet, and can train a VGG-11 model to the same accuracy as end-to-end learning. To our knowledge, this is the first competitive alternative to end-to-end training of CNNs that can scale to ImageNet. We illustrate several interesting properties of these models theoretically and conduct a range of experiments to study the properties this training induces on the intermediate representations.

show abstract

“…One of the reasons why GCNs lack interpretability is because no training objective is assigned to a specific layer except the final one: end-to-end training makes their analysis difficult [34]. They also tend to oversmooth graph representations [47], because applying successively an averaging operator leads to smoother representations.…”

Section: Introductionmentioning

confidence: 99%

Interferometric Graph Transform for Community Labeling

Grinsztajn¹,

Leconte²,

Preux³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

We present a new approach for learning unsupervised node representations in community graphs. We significantly extend the Interferometric Graph Transform (IGT) to community labeling: this non-linear operator iteratively extracts features that take advantage of the graph topology through demodulation operations. An unsupervised feature extraction step cascades modulus non-linearity with linear operators that aim at building relevant invariants for community labeling. Via a simplified model, we show that the IGT concentrates around the E-IGT: those two representations are related through some ergodicity properties. Experiments on community labeling tasks show that this unsupervised representation achieves performances at the level of the state of the art on the standard and challenging datasets Cora, Citeseer, Pubmed and WikiCS. * Equal contributions.Preprint. Under review.

show abstract

Building a Regular Decision Boundary with Deep Networks

Abstract: In this work, we build a generic architecture of Convolutional Neural

Cited by 12 publications

References 29 publications

Fixed-point algorithms for inverse of residual rectifier neural networks

Fixed-point algorithms for inverse of residual rectifier neural networks

Greedy Layerwise Learning Can Scale to ImageNet

Interferometric Graph Transform for Community Labeling

Contact Info

Product

Resources

About