Iterative Denoising using Jensen-Renyi Divergences with an Application to Unsupervised Document Categorization

Karakos, Damianos; Khudanpur, Sanjeev; Eisner, Jason; Priebe, Carey E.

doi:10.1109/icassp.2007.366284

Cited by 6 publications

(3 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We instead prefer a tunable version of mutual information, the Jensen-Rényi divergence (as used in [10], [11]); the tunable parameter allows greater flexibility and more robustness to sparsity (which can yield state-of-the-art results in document categorization [10], [11]). …”

Section: Unsupervised Clustering Algorithms a Unsupervised Ispdtsmentioning

confidence: 99%

Bootstrapping a spoken language identification system using unsupervised integrated sensing and processing decision trees

Huang

Karakos

Coppersmith

et al. 2011

2011 IEEE Workshop on Automatic Speech Recognition &Amp; Understanding

Self Cite

View full text Add to dashboard Cite

Abstract-In many inference and learning tasks, collecting large amounts of labeled training data is time consuming and expensive, and oftentimes impractical. Thus, being able to efficiently use small amounts of labeled data with an abundance of unlabeled data-the topic of semi-supervised learning (SSL) [1]-has garnered much attention. In this paper, we look at the problem of choosing these small amounts of labeled data, the first step in a bootstrapping paradigm. Contrary to traditional active learning where an initial trained model is employed to select the unlabeled data points which would be most informative if labeled, our selection has to be done in an unsupervised way, as we do not even have labeled data to train an initial model.We propose using unsupervised clustering algorithms, in particular integrated sensing and processing decision trees (ISPDTs) [2], to select small amounts of data to label and subsequently use in SSL (e.g. transductive SVMs). In a language identification task on the CallFriend 1 and 2003 NIST Language Recognition Evaluation corpora [3], we demonstrate that the proposed method results in significantly improved performance over random selection of equivalently sized training data.

show abstract

Section: Unsupervised Clustering Algorithms a Unsupervised Ispdtsmentioning

confidence: 99%

Bootstrapping a spoken language identification system using unsupervised integrated sensing and processing decision trees

Huang

Karakos

Coppersmith

et al. 2011

2011 IEEE Workshop on Automatic Speech Recognition &Amp; Understanding

Self Cite

View full text Add to dashboard Cite

show abstract

“…The JRD has been used in several signal/image processing applications, such as registration, segmentation, denoising, and classification [30], [31], [32].…”

Section: The Jensen-rényi Divergencementioning

confidence: 99%

Nonextensive Generalizations of the Jensen-Shannon Divergence

Martins,

Aguiar,

Figueiredo

2008

Preprint

View full text Add to dashboard Cite

Convexity is a key concept in information theory, namely via the many implications of Jensen's inequality, such as the non-negativity of the Kullback-Leibler divergence (KLD). Jensen's inequality also underlies the concept of Jensen-Shannon divergence (JSD), which is a symmetrized and smoothed version of the KLD. This paper introduces new JSD-type divergences, by extending its two building blocks: convexity and Shannon's entropy. In particular, a new concept of q-convexity is introduced and shown to satisfy a Jensen's q-inequality. Based on this Jensen's q-inequality, the Jensen-Tsallis q-difference is built, which is a nonextensive generalization of the JSD, based on Tsallis entropies. Finally, the Jensen-Tsallis q-difference is charaterized in terms of convexity and extrema.

show abstract

“…The JR divergence has been used in signal processing applications (Karakos et al, 2007). We show in Sect.…”

Section: Jensen-shannon (Js) Divergence Consider a Classification Promentioning

confidence: 99%

Nonextensive entropic kernels

Martins

Figueiredo

Aguiar

et al. 2008

Proceedings of the 25th International Conference on Machine Learning - ICML '08

View full text Add to dashboard Cite

Positive definite kernels on probability measures have been recently applied in structured data classification problems. Some of these kernels are related to classic information theoretic quantities, such as mutual information and the Jensen-Shannon divergence. Meanwhile, driven by recent advances in Tsallis statistics, nonextensive generalizations of Shannon's information theory have been proposed. This paper bridges these two trends. We introduce the Jensen-Tsallis q-difference, a generalization of the Jensen-Shannon divergence. We then define a new family of nonextensive mutual information kernels, which allow weights to be assigned to their arguments, and which includes the Boolean, Jensen-Shannon, and linear kernels as particular cases. We illustrate the performance of these kernels on text categorization tasks.

show abstract

Iterative Denoising using Jensen-Renyi Divergences with an Application to Unsupervised Document Categorization

Cited by 6 publications

References 17 publications

Bootstrapping a spoken language identification system using unsupervised integrated sensing and processing decision trees

Bootstrapping a spoken language identification system using unsupervised integrated sensing and processing decision trees

Nonextensive Generalizations of the Jensen-Shannon Divergence

Nonextensive entropic kernels

Contact Info

Product

Resources

About