Learning dynamics and robustness of vector quantization and neural gas

Abstract

Various alternatives have been developed to improve the Winner-Takes-All (WTA) mechanism in vector quantization, including the Neural Gas (NG). However, the behavior of these algorithms including their learning dynamics, robustness with respect to initialization, asymptotic results, etc. has only partially been studied in a rigorous mathematical analysis. The theory of on-line learning allows for an exact mathematical description of the training dynamics in model situations. We demonstrate using a system of three competing prototypes trained from a mixture of Gaussian clusters that the Neural Gas can improve convergence speed and achieves robustness to initial conditions. However, depending on the structure of the data, the Neural Gas does not always obtain the best asymptotic quantization error.

Phase transitions in vector quantization and neural gas

“…insensitive with respect to initial conditions in practice. This gives NG an advantage over WTA schemes, analogous to findings for on-line NG algorithms [20].…”

“…[13]. The description of hei x in terms of order parameters fR ij ; Q kl g can be found in [3,4] for two-prototype systems, and [20] for threeprototype systems. We introduce the entropy term s which represents the phase space volume corresponding to a particular setting of order parameters.…”

“…Densities of the above or a similar form have been studied previously in the context of both supervised and unsupervised learning, see e.g. [4,7,10,20]. Note that for large N, the highly overlapping clusters become only apparent in subspaces that have significant overlap with the B i .…”

“…see [4,20] for the result and details of the calculation. It can be performed analytically for systems with two prototypes and involves numerical Gaussian integrals for KX3.…”

“…[4,19]. Here, we consider training from a set of examples by means of off-line or batch stochastic optimization of a cost function.…”