Auto-association by multilayer perceptrons and singular value decomposition

Bourlard, Hervé; Kamp, Y.

doi:10.1007/bf00332918

Cited by 1,175 publications

(640 citation statements)

References 6 publications

Supporting

Mentioning

633

Contrasting

Unclassified

Order By: Relevance

“…A hidden layer in each part enables the network to perform nonlinear mapping functions. Without these hidden layers, the network would only be able to perform linear PCA even with nonlinear units in the component layer, as shown by Bourlard and Kamp [29]. To regularise the network, a weight decay term is added E total = E + ν i w 2 i in order to penalise large network weights w. In most experiments, ν = 0.001 was a reasonable choice.…”

Section: Standard Nonlinear Pcamentioning

confidence: 99%

Nonlinear Principal Component Analysis: Neural Network Models and Applications

Scholz

Fraunholz

Selbig

2008

Lecture Notes in Computational Science and Enginee

133

View full text Add to dashboard Cite

Summary.Nonlinear principal component analysis (NLPCA) as a nonlinear generalisation of standard principal component analysis (PCA) means to generalise the principal components from straight lines to curves. This chapter aims to provide an extensive description of the autoassociative neural network approach for NLPCA. Several network architectures will be discussed including the hierarchical, the circular, and the inverse model with special emphasis to missing data. Results are shown from applications in the field of molecular biology. This includes metabolite data analysis of a cold stress experiment in the model plant Arabidopsis thaliana and gene expression analysis of the reproductive cycle of the malaria parasite Plasmodium falciparum within infected red blood cells.

show abstract

Section: Standard Nonlinear Pcamentioning

confidence: 99%

Nonlinear Principal Component Analysis: Neural Network Models and Applications

Scholz

Fraunholz

Selbig

2008

Lecture Notes in Computational Science and Enginee

133

View full text Add to dashboard Cite

show abstract

“…Standard optimization methods, such as the gradient descent algorithm, can be used for estimating the network weights. It has been shown that a three-layer autoencoder with a bottleneck in the second layer performs a linear dimensionality reduction specified by the principal components of the input distribution (Bourlard & Kamp, 1988;Baldi & Hornik, 1989;Oja, 1991). Specifically, if an autoencoder containing M hidden units is trained by minimizing the error function (17), the network projects the data onto the M-dimensional sub-space spanned by the first M principal components of the data.…”

Section: Linear and Non-linear Autoencoders For Dimensionality Reductionmentioning

confidence: 99%

Unsupervised visual learning of three-dimensional objects using a modular network architecture

1999

View full text Add to dashboard Cite

This paper presents a modular network architecture that learns to cluster multiple views of multiple three-dimensional (3D) objects. The proposed network model is based on a mixture of non-linear autoencoders, which compete to encode multiple views of each 3D object. The main advantage of using a mixture of autoencoders is that it can capture multiple non-linear sub-spaces, rather than multiple centers for describing complex shapes of the view distributions. The unsupervised training algorithm is formulated within a maximum-likelihood estimation framework. The performance of the modular network model is evaluated through experiments using synthetic 3D wire-frame objects and gray-level images of real 3D objects. It is shown that the performance of the modular network model is superior to the performance of the conventional clustering algorithms, such as the K-means algorithm and the Gaussian mixture model. ᭧

show abstract

“…Interestingly, such a feature-generating system already exists and in fact has become quite popular through the rise of deep learning (Bengio et al, 2007;Hinton et al, 2006;Le et al, 2012;Marc'Aurelio et al, 2007): the autoencoder (Hinton and Zemel, 1994;Bourlard and Kamp, 1988). Autoencoders, which can be trained through a variety of algorithms from Restricted Boltzmann Machines (RBMs) (Hinton et al, 2006) to more conventional stochastic gradient descent (Le et al, 2012) (e.g.…”

Section: Introductionmentioning

confidence: 99%

Real-time Hebbian Learning from Autoencoder Features for Control Tasks

Pugh

Soltoggio

Stanley

2014

Artificial Life 14: Proceedings of the Fourteenth International Conference on the Synthesis and Simulation of Living Systems

View full text Add to dashboard Cite

Neural plasticity and in particular Hebbian learning play an important role in many research areas related to artficial life. By allowing artificial neural networks (ANNs) to adjust their weights in real time, Hebbian ANNs can adapt over their lifetime. However, even as researchers improve and extend Hebbian learning, a fundamental limitation of such systems is that they learn correlations between preexisting static features and network outputs. A Hebbian ANN could in principle achieve significantly more if it could accumulate new features over its lifetime from which to learn correlations. Interestingly, autoencoders, which have recently gained prominence in deep learning, are themselves in effect a kind of feature accumulator that extract meaningful features from their inputs. The insight in this paper is that if an autoencoder is connected to a Hebbian learning layer, then the resulting Realtime Autoencoder-Augmented Hebbian Network (RAAHN) can actually learn new features (with the autoencoder) while simultaneously learning control policies from those new features (with the Hebbian layer) in real time as an agent experiences its environment. In this paper, the RAAHN is shown in a simulated robot maze navigation experiment to enable a controller to learn the perfect navigation strategy significantly more often than several Hebbian-based variant approaches that lack the autoencoder. In the long run, this approach opens up the intriguing possibility of real-time deep learning for control.

show abstract

Auto-association by multilayer perceptrons and singular value decomposition

Cited by 1,175 publications

References 6 publications

Nonlinear Principal Component Analysis: Neural Network Models and Applications

Nonlinear Principal Component Analysis: Neural Network Models and Applications

Unsupervised visual learning of three-dimensional objects using a modular network architecture

Real-time Hebbian Learning from Autoencoder Features for Control Tasks

Contact Info

Product

Resources

About