Convergence of Stochastic Vector Quantization and Learning Vector Quantization with Bregman Divergences

Mavridis, Christos N.; Baras, John S.

doi:10.1016/j.ifacol.2020.12.006

Cited by 12 publications

(5 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This follows from the fact that as T → 0, it degenerates to the stochastic vector quantization algorithm (Def. 1), the consistency of which is already proven [10], [26].…”

Section: − − → E [X|µ]supporting

confidence: 52%

“…The consistency of this classifier at the limit T → 0, can be shown with arguments similar to these of the LVQ algorithm [26].…”

Section: B Online Deterministic Annealing For Classificationmentioning

confidence: 77%

“…When the training data are not available a priori but are being observed online, or when the processing of the entire dataset in every iteration is computationally expensive, a stochastic vector quantization algorithm can be defined as a recursive asynchronous stochastic approximation algorithm [26]:…”

Section: A Vector Quantization For Clusteringmentioning

confidence: 99%

“…AND SUPERVISED LEARNING Online vector quantization algorithms, are proven to converge to locally optimal configurations [26]. However, similar to common neural networks, their convergence properties and final configuration depend heavily on two design parameters: (a) the number of clusters (neurons), and (b) their initial configuration.…”

Section: Deterministic Annealing For Unsupervisedmentioning

confidence: 99%

“…As a result it can be slow to converge, loops over the entire dataset multiple times, and cannot be used with very large datasets or online observations. While vector quantization algorithms can be formulated as online stochastic gradient descent methods [9], [26], the same has not been shown, as of yet, for deterministic annealing, despite the fact that its formulation is known for decades. In this regard, designing an online learning rule for deterministic annealing could lead to new developments, similar to the introduction of a greedy online training algorithm for deep belief networks [2].…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Online Deterministic Annealing for Classification and Clustering

Mavridis,

Baras

2021

Preprint

View full text Add to dashboard Cite

We introduce an online prototype-based learning algorithm for clustering and classification, based on the principles of deterministic annealing. We show that the proposed algorithm constitutes a competitive-learning neural network, the learning rule of which is formulated as an online stochastic approximation algorithm. The annealing nature of the algorithm prevents poor local minima, offers robustness with respect to the initial conditions, and provides a means to progressively increase the complexity of the learning model as needed, through an intuitive bifurcation phenomenon. As a result, the proposed approach is interpretable, requires minimal hyper-parameter tuning, and offers online control over the complexity-accuracy trade-off. Finally, Bregman divergences are used as a family of dissimilarity measures that are shown to play an important role in both the performance of the algorithm, and its computational complexity. We illustrate the properties and evaluate the performance of the proposed learning algorithm in artificial and real datasets.

show abstract

“…This follows from the fact that as T → 0, it degenerates to the stochastic vector quantization algorithm (Def. 1), the consistency of which is already proven [10], [26].…”

Section: − − → E [X|µ]supporting

confidence: 52%

“…The consistency of this classifier at the limit T → 0, can be shown with arguments similar to these of the LVQ algorithm [26].…”

Section: B Online Deterministic Annealing For Classificationmentioning

confidence: 77%

Section: A Vector Quantization For Clusteringmentioning

confidence: 99%

Section: Deterministic Annealing For Unsupervisedmentioning

confidence: 99%