Deterministic annealing for clustering, compression, classification, regression, and related optimization problems

Abstract

The deterministic annealing approach to clustering and its extensions has demonstrated substantial performance improvement over standard supervised and unsupervised learning methods in a variety of important applications including compression, estimation, pattern recognition and classification, and statistical regression. The method offers three important features: 1) the ability to avoid many poor local optima; 2) applicability to many different structures/architectures; and 3) the ability to minimize the right cost function even when its gradients vanish almost everywhere, as in the case of the empirical classification error. It is derived within a probabilistic framework from basic information theoretic principles (e.g., maximum entropy and random coding). The application-specific cost is minimized subject to a constraint on the randomness (Shannon entropy) of the solution, which is gradually lowered. We emphasize intuition gained from analogy to statistical physics, where this is an annealing process that avoids many shallow local minima of the specified cost and, at the limit of zero "temperature," produces a nonrandom (hard) solution. Alternatively, the method is derived within rate-distortion theory, where the annealing process is equivalent to computation of Shannon's rate-distortion function, and the annealing temperature is inversely proportional to the slope of the curve. This provides new insights into the method and its performance, as well as new insights into rate-distortion theory itself. The basic algorithm is extended by incorporating structural constraints to allow optimization of numerous popular structures including vector quantizers, decision trees, multilayer perceptrons, radial basis functions, and mixtures of experts. Experimental results show considerable performance gains over standard structure-specific and application-specific training methods. The paper concludes with a brief discussion of extensions of the method that are currently under investigation.

Deterministic annealing as a jet clustering algorithm in hadronic collisions

“…Other choices are possible. For example one could take into account that the phase transition producing the splitting of a cluster occurs at a temperature proportional to the variance of the cluster itself [4]. So a characteristic of well defined clusters is that they are stable for a wide range of temperature and this stability property could be used in jet recognition.…”

“…The word deterministic refers to the fact that, as we shall see, thermal equilibrium is obtained minimizing directly the free energy, in opposition to the stochastic simulation used by Simulated Annealing [14]. We introduce here a formulation of DA called Mass-Constrained Clustering (MCC) [3,4] that is particularly suitable for our application. In effect in this formulation the number of clusters is not fixed a priori, as it happened in the precedent application of DA to the jet searching problem [1], but is the result of the calculation.…”

“…This task is performed minimizing a prescribed cost function that is to be adapted to the problem under investigation. In a recent paper [1] it has been shown that a particular clustering algorithm, the so-called Deterministic Annealing (DA) [2,3,4], can be adapted to the study of the hadronic jets in high energy e + e − scattering. Essentially, DA can give the same results of the standard Durham algorithm in a faster way, as a consequence of a lower computational complexity.…”

On the Number of Modes of a Gaussian Mixture

“…Thus, the new point x (τ +1) is the conditional mean of the mixture under the current point x (τ ) . This is formally akin to clustering by deterministic annealing (Rose, 1998), to algorithms for finding pre-images in kernel-based methods (Schölkopf et al, 1999) and to mean-shift algorithms (section 5.2).…”

Annealing and the normalized N-cut

“…Here the emphasis is on the control of the quality of the solution via a selection of the annealing parameter. The solution "emerges" gradually as a function of this parameter, rather then being computed at once [11][12][13]. A special class of annealing problems involve information distortion type cost functions [15,16,12,13] which have been applied to clustering problems in neuroscience, image processing, spectral analysis, gene expression, stock prices, and movie ratings [14,19,17,18].…”