Optimal unsupervised learning

Watkin, T. L. H.; Nadal, Jean-Pierre

doi:10.1088/0305-4470/27/6/016

Cited by 50 publications

(84 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…orthogonal space are reflected by non-trivial configurations of fQ ij g. The underlying cluster structure is not at all detected as long as e a is smaller than the critical value e a c . This parallels findings for supervised learning in neural networks with two hidden units [5] or unsupervised learning scenarios [10,16]. Above e a c , prototypes begin to align with the clusters and the system becomes specialized, i.e.…”

Section: Specialization Transition In the Training Processsupporting

confidence: 72%

See 1 more Smart Citation

Phase transitions in vector quantization and neural gas

Witoelar

Biehl

2009

Neurocomputing

View full text Add to dashboard Cite

a b s t r a c tThe statistical physics of off-learning is applied to winner-takes-all (WTA) and rank-based vector quantization (VQ), including the neural gas (NG). The analysis is based on the limit of high training temperatures and the annealed approximation. The typical learning behavior is evaluated for systems of two and three prototypes with data drawn from a mixture of high dimensional Gaussian clusters. The learning curves exhibit phase transitions, i.e. a critical or discontinuous dependence of performances on the training set size and training temperature. We show how the nature and properties of the transition depend on the number of prototypes and the control parameter of rank-based cost functions. The NGbased systems are demonstrated to give an advantage over WTA in terms of robustness to initial conditions.

show abstract

Section: Specialization Transition In the Training Processsupporting

confidence: 72%

“…Similar effects of ''retarded learning'' have been studied in several models and learning scenarios earlier, e.g. [5,6,8,10,16].…”

Section: Introductionmentioning

confidence: 59%

Phase transitions in vector quantization and neural gas

Witoelar

Biehl

2009

Neurocomputing

View full text Add to dashboard Cite

show abstract

“…This task of identifying signal-carrying principal components has been analyzed in statistical physics, where the phenomenon of “retarded classification” was described (Watkin and Nadal, 1994). Let α be the ratio of the number of examples to the number of variables.…”

Section: Discussionmentioning

confidence: 99%

“…A phase transition happens at this point, and, if this level has not been reached, it is impossible to identify the signal-carrying components in the noisy data although good signal detection is still possible under some circumstances (Watkin and Nadal, 1994; see also Results and Discussion below).…”

Section: Introductionmentioning

confidence: 99%

Dimensionality estimation for optimal detection of functional networks in BOLD fMRI data

et al. 2011

View full text Add to dashboard Cite

Estimation of the intrinsic dimensionality of fMRI data is an important part of data analysis that helps to separate the signal of interest from noise. We have studied multiple methods of dimensionality estimation proposed in the literature and used these estimates to select a subset of principal components that was subsequently processed by linear discriminant analysis (LDA). Using simulated multivariate Gaussian data, we show that the dimensionality that optimizes signal detection (in terms of the receiver operating characteristic (ROC) metric) goes through a transition from many dimensions to a single dimension as a function of the signal-to-noise ratio. This transition happens when the loci of activation are organized into a spatial network and the variance of the networked, task-related signals is high enough for the signal to be easily detected in the data. We show that reproducibility of activation maps is a metric that captures this switch in intrinsic dimensionality. Except for reproducibility, all of the methods of dimensionality estimation we considered failed to capture this transition: optimization of Bayesian evidence, minimum description length, supervised and unsupervised LDA prediction, and Stein's unbiased risk estimator. This failure results in sub-optimal ROC performance of LDA in the presence of a spatially distributed network, and may have caused LDA to underperform in many of the reported comparisons in the literature. Using real fMRI data sets, including multi-subject group and within-subject longitudinal analysis we demonstrate the existence of these dimensionality transitions in real data.

show abstract

“…(From Biehl 1997) Another possibility is to select a direction W according to the a posteriori distribution, Eq. (35), for instance by using the Monte Carlo method [Watkin and Nadal 1994]. The result is obtained from calculating Z for β = 1, again by averaging ln Z over the true distribution of data points.…”

mentioning

confidence: 99%

Phase transitions of neural networks

Kinzel

1998

Philosophical Magazine B

View full text Add to dashboard Cite

The cooperative behaviour of interacting neurons and synapses is studied using models and methods from statistical physics. The competition between training error and entropy may lead to discontinuous properties of the neural network. This is demonstrated for a few examples: Perceptron, associative memory, learning from examples, generalization, multilayer networks, structure recognition, Bayesian estimate, on-line training, noise estimation and time series generation.

show abstract

Optimal unsupervised learning

Cited by 50 publications

References 11 publications

Phase transitions in vector quantization and neural gas

Phase transitions in vector quantization and neural gas

Dimensionality estimation for optimal detection of functional networks in BOLD fMRI data

Phase transitions of neural networks

Contact Info

Product

Resources

About