Information theoretic clustering

Abstract

Clustering is one of the important topics in pattern recognition. Since only the structure of the data dictates the grouping (unsupervised learning), information theory is an obvious criteria to establish the clustering rule. This paper describes a novel valley seeking clustering algorithm using an information theoretic measure to estimate the cost of partitioning the data set. The information theoretic criteria developed here evolved from a Renyi's entropy estimator that was proposed recently and has been successfully applied to other machine learning applications. An improved version of the k-change algorithm is used in optimization because of the stepwise nature of the cost function and existence of local minima. Even when applied to nonlinearly separable data, the new algorithm performs well, and was able to find nonlinear boundaries between clusters. The algorithm is also applied to the segmentation of magnetic resonance imaging data (MRI) with very promising results.

Information Theoretic Clustering Using Minimum Spanning Trees

“…k} to these samples. We adopt the viewpoint of information theoretic clustering of Gokcay and Principe [6], where the x i are considered i.i.d. samples from a distribution p(X), and the y i are found such that the mutual information I(X, Y ) between the distribution p(X) and the assigned labels p(Y ) is maximized.…”

Generalized Clustering via Kernel Embeddings

“…Informationtheoretic clustering (Gokcay andPrincipe, 2002, Jenssen et al, 2004) measures discriminability by the cosine of the angle between the µ[P k ]. Its motivation, however, restricts k to have a speci c form, whereas MMD is more general.…”

Identifying the components

“…Entropy measures allow for non-linearly defined clusters to be found in image segmentation tasks (Gokcay and Principe 2002). The MDL principle was used for vector quantization (Bischof et al 1999), where superfluous vectors were detected via MDL.…”