Sparse Coding and Autoencoders

Rangamani, Akshay; Mukherjee, Anirbit; Basu, Amitabh; Ganapathy, Tejaswini; Arora, Ashish; Chin, Sang; Tran, Trac D.

doi:10.48550/arxiv.1708.03735

Cited by 2 publications

(11 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We conclude with the case of training two-layer autoencoders whose weights are shared (i.e., A = W ). This is a common architectural choice in practice, and indeed previous theoretical analysis for autoencoders [Rangamani et al, 2017, Nguyen et al, 2019, Li and Phan-Minh, 2019 have focused on this setting. We will show that somewhat surprisingly, allowing the network to be over-parameterized in this setting leads to certain degeneracies.…”

Section: Weight-tied Autoencodersmentioning

confidence: 99%

“…None of the above papers have rigorously studied the training dynamics of autoencoder models. The loss surface of autoencoder training was first characterized in Rangamani et al [2017]. Subsequently, Nguyen et al [2019] proved that under-parameterized (and suitably initialized) autoencoders performed (approximate) proper parameter learning in the regime of asymptotically many samples, building upon techniques in provable dictionary learning; cf.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Benefits of Jointly Training Autoencoders: An Improved Neural Tangent Kernel Analysis

Nguyen¹,

Wong²,

Hegde³

2019

Preprint

View full text Add to dashboard Cite

A remarkable recent discovery in machine learning has been that deep neural networks can achieve impressive performance (in terms of both lower training error and higher generalization capacity) in the regime where they are massively over-parameterized. Consequently, over the last several months, the community has devoted growing interest in analyzing optimization and generalization properties of over-parameterized networks, and several breakthrough works have led to important theoretical progress. However, the majority of existing work only applies to supervised learning scenarios and hence are limited to settings such as classification and regression.In contrast, the role of over-parameterization in the unsupervised setting has gained far less attention. In this paper, we study the gradient dynamics of two-layer over-parameterized autoencoders with ReLU activation. We make very few assumptions about the given training dataset (other than mild nondegeneracy conditions). Starting from a randomly initialized autoencoder network, we rigorously prove the linear convergence of gradient descent in two learning regimes, namely:1. the weakly-trained regime where only the encoder is trained, and 2. the jointly-trained regime where both the encoder and the decoder are trained.Our results indicate the considerable benefits of joint training over weak training for finding global optima, achieving a dramatic decrease in the required level of over-parameterization.We also analyze the case of weight-tied autoencoders (which is a commonly used architectural choice in practical settings) and prove that in the over-parameterized setting, training such networks from randomly initialized points leads to certain unexpected degeneracies.

show abstract

Section: Weight-tied Autoencodersmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Benefits of Jointly Training Autoencoders: An Improved Neural Tangent Kernel Analysis

Nguyen¹,

Wong²,

Hegde³

2019

Preprint

View full text Add to dashboard Cite

show abstract

“…Our focus here is on shallow two-layer autoencoder architectures with shared weights. Conceptually, we build upon previous theoretical results on learning autoencoder networks [Arora et al, 2014a, 2015a, Rangamani et al, 2017, and we elaborate on the novelty of our work in the discussion on prior work below.…”

Section: Our Contributionsmentioning

confidence: 99%

“…samples from a high-dimensional distribution parameterized by a generative model, and we train the weights of the autoencoder using ordinary (batch) gradient descent. We consider three families of generative models that are commonly adopted in machine learning: (i) the Gaussian mixture model with well-separated centers [Arora and Kannan, 2005]; (ii) the k-sparse model, specified by sparse linear combination of atoms [Spielman et al, 2012]; and (iii) the non-negative k-sparse model [Rangamani et al, 2017]. While these models are traditionally studied separately depending on the application, all of these model families can be expressed via a unified, generic form:…”

Section: Our Contributionsmentioning

confidence: 99%

“…More recently, autoencoders have been shown to learn sparse representations [Arpit et al, 2015]. The recent paper [Rangamani et al, 2017] demonstrates that under the sparse generative model, the standard squared-error reconstruction loss of ReLU autoencoders exhibits (with asymptotically many samples) critical points in a neighborhood of the ground truth dictionary. However, they do not analyze gradient dynamics, nor do they establish convergence rates.…”

Section: Comparison With Prior Workmentioning

confidence: 99%

See 1 more Smart Citation

Autoencoders Learn Generative Linear Models

Nguyen,

Wong,

Hegde

2018

Preprint

View full text Add to dashboard Cite

We provide a series of results for unsupervised learning with autoencoders. Specifically, we study shallow two-layer autoencoder architectures with shared weights. We focus on three generative models for data that are common in statistical machine learning: (i) the mixture-of-gaussians model, (ii) the sparse coding model, and (iii) the sparsity model with non-negative coefficients. For each of these models, we prove that under suitable choices of hyperparameters, architectures, and initialization, autoencoders learned by gradient descent can successfully recover the parameters of the corresponding model. To our knowledge, this is the first result that rigorously studies the dynamics of gradient descent for weightsharing autoencoders. Our analysis can be viewed as theoretical evidence that shallow autoencoder modules indeed can be used as feature learning mechanisms for a variety of data models, and may shed insight on how to train larger stacked architectures with autoencoders as basic building blocks.

show abstract

Sparse Coding and Autoencoders

Cited by 2 publications

References 0 publications

Benefits of Jointly Training Autoencoders: An Improved Neural Tangent Kernel Analysis

Benefits of Jointly Training Autoencoders: An Improved Neural Tangent Kernel Analysis

Autoencoders Learn Generative Linear Models

Contact Info

Product

Resources

About