Learning From Hidden Traits: Joint Factor Analysis and Latent Clustering

Abstract

Dimensionality reduction techniques play an essential role in data analytics, signal processing and machine learning. Dimensionality reduction is usually performed in a preprocessing stage that is separate from subsequent data analysis, such as clustering or classification. Finding reduced-dimension representations that are well-suited for the intended task is more appealing. This paper proposes a joint factor analysis and latent clustering framework, which aims at learning cluster-aware low-dimensional representations of matrix and tensor data. The proposed approach leverages matrix and tensor factorization models that produce essentially unique latent representations of the data to unravel latent cluster structure -- which is otherwise obscured because of the freedom to apply an oblique transformation in latent space. At the same time, latent cluster structure is used as prior information to enhance the performance of factorization. Specific contributions include several custom-built problem formulations, corresponding algorithms, and discussion of associated convergence properties. Besides extensive simulations, real-world datasets such as Reuters document data and MNIST image data are also employed to showcase the effectiveness of the proposed approaches

K-MACE and Kernel K-MACE Clustering

“…Values of m, the actual number of clusters, d, the dimension of the data, and N , the number of samples in the data set are provided in the close to 5 by K-MACE. The reason is the structure of the data for which the number of elements, n mj , of each cluster are [143,77,52,35,20,5,2,2]. As these numbers show, the last three clusters have very few elements that can't be detected as independent clusters by K-MACE methods.…”

“…Similar to other partitional approaches, K-means requires the correct number of clusters (CNC) to finalize the clustering procedure. In general, most clustering algorithms require CNC estimate as their input [4], [5], [6], [7]. However, in many practical applications this value is not available and CNC has to be estimated during the clustering procedure by using the same data that requires clustering.…”

On Recoverability of Randomly Compressed Tensors with Low CP Rank

“…The sample complexity proved in [21] is appealing, which is exactly the number of unknowns. The caveat is that the A (n) 's have to follow a certain continuous distribution-which means that some important types of tensors (e.g., tensors with discrete latent factors that have applications in machine learning [18,26,27]) may not be covered by the recoverability theorem in [21].…”

Clustering With Orthogonal AutoEncoder

“…NLS3C [18] iteratively updates the sparse similarity matrix and clustering labels via an efficient manner. JNKM [19] jointly incorporates non-negative matrix factorization DR algorithm and K-means clustering algorithm. RCC [20] performs DR stage and clustering stage together by optimizing a continuous global objective.…”