Lower and upper bounds for approximation of the Kullback-Leibler divergence between Gaussian Mixture Models

Durrieu, Jean‐Louis; Thiran, Jean-Philippe; Kelly, Finnian

doi:10.1109/icassp.2012.6289001

Cited by 61 publications

(62 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While our work formulates the KL divergence with the same HMM definitions as [7], and extends the variational approximation to the case of HMMs with Gaussian mixture models (GMMs) modelling the observation probabilities at states. Moreover, our work can be considered as the expansions of the methods for approximating the KL divergence between GMMs in [8] and [9]. We finally confirm the effectiveness of the proposed methods by experiments.…”

Section: Introductionsupporting

confidence: 55%

See 1 more Smart Citation

Upper and lower bounds for approximation of the Kullback-Leibler divergence between Hidden Markov models

Han

Zheng

et al. 2013

2013 IEEE International Conference on Acoustics, Speech and Signal Processing

View full text Add to dashboard Cite

The Kullback-Leibler (KL) divergence is often used for a similarity comparison between two Hidden Markov models (HMMs). However, there is no closed form expression for computing the KL divergence between HMMs, and it can only be approximated. In this paper, we propose two novel methods for approximating the KL divergence between the left-to-right transient HMMs. The first method is a product approximation which can be calculated recursively without introducing extra parameters. The second method is based on the upper and lower bounds of KL divergence, and the mean of these bounds provides an available approximation of the divergence. We demonstrate the effectiveness of the proposed methods through experiments including the deviations to the numerical approximation and the task of predicting the confusability of phone pairs. Experimental resuls show that the proposed product approximation is comparable with the current variational approximation, and the proposed approximation based on bounds performs better than current methods in the experiments.

show abstract

Section: Introductionsupporting

confidence: 55%

“…In [9], an approximation of the KL divergence between GMMs are adopted based on the idea that strict bounds can provide an interval in which the real value of the KL divergence can be found. Motivated by this idea, we also design an approximation for the divergence between HMMs based on bounds.…”

Section: Approximation Based On Bounds For the Kl Divergencementioning

confidence: 99%

Upper and lower bounds for approximation of the Kullback-Leibler divergence between Hidden Markov models

Han

Zheng

et al. 2013

2013 IEEE International Conference on Acoustics, Speech and Signal Processing

View full text Add to dashboard Cite

show abstract

“…Similarly, in the case of Kullback-Leibler divergence, an analytic evaluation of the differential entropy is also impossible. Thus, approximate calculations become inevitable [7][8][9]. Jenssen et al [3] use the Rényi entropy [10] as a similarity measure between clusters.…”

Section: Introductionmentioning

confidence: 99%

Bounds on Rényi and Shannon Entropies for Finite Mixtures of Multivariate Skew-Normal Distributions: Application to Swordfish (Xiphias gladius Linnaeus)

Contreras-Reyes

Cortés

2016

Entropy

View full text Add to dashboard Cite

Mixture models are in high demand for machine-learning analysis due to their computational tractability, and because they serve as a good approximation for continuous densities. Predominantly, entropy applications have been developed in the context of a mixture of normal densities. In this paper, we consider a novel class of skew-normal mixture models, whose components capture skewness due to their flexibility. We find upper and lower bounds for Shannon and Rényi entropies for this model. Using such a pair of bounds, a confidence interval for the approximate entropy value can be calculated. In addition, an asymptotic expression for Rényi entropy by Stirling's approximation is given, and upper and lower bounds are reported using multinomial coefficients and some properties and inequalities of L p metric spaces. Simulation studies are then applied to a swordfish (Xiphias gladius Linnaeus) length dataset.

show abstract

“…The dissimilarity arises in the condition in (47) for Lemma 5, which is different from the condition in (41) for Lemma 4. This changes the range for the minimax risk ε * in which the lower bound in (45) holds.…”

Section: A Sparse Gaussian Coefficientsmentioning

confidence: 99%

Minimax Lower Bounds on Dictionary Learning for Tensor Data

Shakeri

Bajwa

Sarwate

2018

IEEE Trans. Inform. Theory

View full text Add to dashboard Cite

This paper provides fundamental limits on the sample complexity of estimating dictionaries for tensor data. The specific focus of this work is on Kth-order tensor data and the case where the underlying dictionary can be expressed in terms of K smaller dictionaries. It is assumed the data are generated by linear combinations of these structured dictionary atoms and observed through white Gaussian noise. This work first provides a general lower bound on the minimax risk of dictionary learning for such tensor data and then adapts the proof techniques for specialized results in the case of sparse and sparse-Gaussian linear combinations. The results suggest the sample complexity of dictionary learning for tensor data can be significantly lower than that for unstructured data: for unstructured data it scales linearly with the product of the dictionary dimensions, whereas for tensor-structured data the bound scales linearly with the sum of the product of the dimensions of the (smaller) component dictionaries. A partial converse is provided for the case of 2ndorder tensor data to show that the bounds in this paper can be tight. This involves developing an algorithm for learning highlystructured dictionaries from noisy tensor data. Finally, numerical experiments highlight the advantages associated with explicitly accounting for tensor data structure during dictionary learning.

show abstract

Lower and upper bounds for approximation of the Kullback-Leibler divergence between Gaussian Mixture Models

Cited by 61 publications

References 5 publications

Upper and lower bounds for approximation of the Kullback-Leibler divergence between Hidden Markov models

Upper and lower bounds for approximation of the Kullback-Leibler divergence between Hidden Markov models

Bounds on Rényi and Shannon Entropies for Finite Mixtures of Multivariate Skew-Normal Distributions: Application to Swordfish (Xiphias gladius Linnaeus)

Minimax Lower Bounds on Dictionary Learning for Tensor Data

Contact Info

Product

Resources

About