Nonlinear higher-order statistical decorrelation by volume-conserving neural architectures

Deco, Gustavo; Brauer, Wilfried

doi:10.1016/0893-6080(94)00108-x

Cited by 107 publications

(49 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The two distinct choices of this gauge invariance are known as Bell-Sejnowski ICA 19 and volumeconserving ICA, 28 respectively (see Appendix D). In the case of volume-conserving ICA, the evaluation of the multi-information difference requires one to estimate only the marginal entropies:…”

Section: ͑5͒mentioning

confidence: 99%

Factorial coding of natural images: how effective are linear models in removing higher-order dependencies?

Bethge

2006

J. Opt. Soc. Am. A

View full text Add to dashboard Cite

(Doc. ID 61982) The performance of unsupervised learning models for natural images is evaluated quantitatively by means of information theory. We estimate the gain in statistical independence (the multi-information reduction) achieved with independent component analysis (ICA), principal component analysis (PCA), zero-phase whitening, and predictive coding. Predictive coding is translated into the transform coding framework, where it can be characterized by the constraint of a triangular filter matrix. A randomly sampled whitening basis and the Haar wavelet are included in the comparison as well. The comparison of all these methods is carried out for different patch sizes, ranging from 2 ϫ 2 to 16ϫ 16 pixels. In spite of large differences in the shape of the basis functions, we find only small differences in the multi-information between all decorrelation transforms (5% or less) for all patch sizes. Among the second-order methods, PCA is optimal for small patch sizes and predictive coding performs best for large patch sizes. The extra gain achieved with ICA is always less than 2%. In conclusion, the edge filters found with ICA lead to only a surprisingly small improvement in terms of its actual objective.

show abstract

Section: ͑5͒mentioning

confidence: 99%

Factorial coding of natural images: how effective are linear models in removing higher-order dependencies?

Bethge

2006

J. Opt. Soc. Am. A

View full text Add to dashboard Cite

show abstract

“…Several techniques have been suggested to capture higher-order statistics from data 14,[16][17][18][19] , but all methods are iterative this requires ad hoc parameter selection, large training set sizes can be needed, and generalization and convergence problems can arise. Thus, our nonlinear noniterative solution is of importance.…”

Section: Introductionmentioning

confidence: 99%

General methodology for simultaneous representation and discrimination of multiple object classes

Talukder

1998

Opt. Eng

View full text Add to dashboard Cite

In this paper we address a new general method for linear and nonlinear feature extraction for simultaneous representation and classi cation. We call this approach the maximum representation and discrimination feature MRDF method. We develop a novel nonlinear eigenfeature extraction NLEF technique to represent data with closed-form solutions and use it to derive a nonlinear MRDF algorithm. Results of the MRDF on synthetic databases are shown and compared with results from standard Fukunaga-Koontz transform and Fisher discriminant function methods. The method is also applied to an automated product inspection problem discrimination and for classi cation and pose estimation of two similar objects under 3-D aspect angle variations representation and discrimination.

show abstract

“…This property is intuitively sound, and is of great use in the derivation of algorithms, such as MISEP, based on the minimization of the mutual information, as we shall see ahead. Mutual information has been used as a criterion for ICA in several different ways (for examples of its use in linear ICA see work by Amari et al 1996;Haykin and Gupta 1999;Almeida 2000b;Taleb and Jutten 1997; for nonlinear ICA examples see work by Yang et al 1998;Deco and Brauer 1995;Almeida 2000a). This paper's central topic is the method of Almeida (2000a,b).…”

Section: H(y I ) − H(y)mentioning

confidence: 99%

“…Nonlinear ICA and nonlinear BSS, on the other hand, are techniques that are still largely under development, and have the potential to become rather powerful tools. Some work on nonlinear ICA has already been published (Burel, 1992;Deco and Brauer, 1995;Marques and Almeida, 1996;Yang et al, 1998;Marques and Almeida, 1999;Palmieri et al, 1999;Valpola, 2000;Almeida, 2000a;Harmeling et al, 2001;Martinez and Bray, 2003).…”

Section: Introductionmentioning

confidence: 99%

“…Some of the dependence measures that have been proposed are based on a quadratic "error" between probability densities (Burel, 1992), on moments of all orders (Marques and Almeida, 1996), on Renyi's entropy (Marques and Almeida, 1999) or on the mutual information of the estimated components (e.g. Deco and Brauer, 1995;Taleb and Jutten, 1997;Almeida, 2000a,b). The latter, the mutual information (MI) of the estimated components, is rather appealing as a dependence measure for several reasons.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Linear and nonlinear ICA based on mutual information

Almeida

Proceedings of the IEEE 2000 Adaptive Systems for Signal Processing, Communications, and Control Symposium (Cat. No.00EX373)

View full text Add to dashboard Cite

Linear Independent Components Analysis (ICA) has become an important signal processing and data analysis technique, the typical application being blind source separation in a wide range of signals, such as biomedical, acoustical and astrophysical ones. Nonlinear ICA is less developed, but has the potential to become at least as powerful.This paper presents MISEP, an ICA technique for linear and nonlinear mixtures, which is based on the minimization of the mutual information of the estimated components. MISEP is a generalization of the popular INFOMAX technique, which is extended in two ways: (1) to deal with nonlinear mixtures, and (2) to be able to adapt to the actual statistical distributions of the sources, by dynamically estimating the nonlinearities to be used at the outputs. The resulting MISEP method optimizes a network with a specialized architecture, with a single objective function: the output entropy.The paper also briefly discusses the issue of nonlinear source separation. Examples of linear and nonlinear source separation performed by MISEP are presented.

show abstract

Nonlinear higher-order statistical decorrelation by volume-conserving neural architectures

Cited by 107 publications

References 12 publications

Factorial coding of natural images: how effective are linear models in removing higher-order dependencies?

Factorial coding of natural images: how effective are linear models in removing higher-order dependencies?

General methodology for simultaneous representation and discrimination of multiple object classes

Linear and nonlinear ICA based on mutual information

Contact Info

Product

Resources

About