Clustering With Orthogonal AutoEncoder

Abstract

Recently, clustering algorithms based on deep AutoEncoder attract lots of attention due to their excellent clustering performance. On the other hand, the success of PCA-Kmeans and spectral clustering corroborates that the orthogonality of embedding is beneficial to increase the clustering accuracy. In this paper, we propose a novel dimensional reduction model, called Orthogonal AutoEncoder (OAE), which encourages the orthogonality of the learned embedding. Furthermore, we propose a joint deep Clustering framework based on Orthogonal AutoEncoder (COAE), and this new framework is capable of extracting the latent embedding and predicting the clustering assignment simultaneously. The COAE stacks a fully connected clustering layer on top of the OAE, where the activation function of the clustering layer is the multinomial logistic regression function. The loss function of the COAE contains two terms: the reconstruction loss and the clustering-oriented loss. The first one is a data-dependent term in order to prevent overfitting. The other one is the cross entropy between the predicted assignment and the auxiliary target distribution. The network parameters of the COAE can be effectively updated by the mini-batch stochastic gradient descent algorithm and the back-propagation approach. The experiments on benchmark datasets empirically demonstrate that the COAE can achieve superior or competitive clustering performance as state-of-the-art deep clustering frameworks.

Dimensionality Reduction via an Orthogonal Autoencoder Approach for Hyperspectral Image Classification

“…However, estimating the network's parameters via the minimization of the reconstruction error L (•) does not guarantee that the features in the latent space are orthogonal to each other, which might provide a better classdiscriminability. Thus, in order to ensure orthogonality among the components in the latent space, we added a regularization term into the reconstruction error L (•), similar to Wang et al (2019). Formally, the orthogonal reconstruction error L (•) can be defined as:…”

“…For example, Chen et al (2014) employed Autoencoders, whereas Liu et al (2018) used Recurrent Neural Networks for hyperspectral image classification tasks. In a similar path, Wang et al (2019) developed a semi-supervised learning method for clustering, which improves class-separability among the latent features via Orthogonal Autoencoders in general classification tasks, however, to the best of our knowledge, Wang's method was never used in the hyperspectral image analysis thus far.…”

“…In this work, we propose a novel semi-supervised approach to reduce the spectral dimensionality of hyperspectral images. Our method, called Hyperspectral Orthogonal Autoencoders (HOAE), extends the work developed by Wang et al (2019) on autoencoders to ensure orthogonality among the features compounding the new lower-dimensional space from hyperspectral image data. To assess our approach, first, we used it to reduce the spectral dimensionality of hyperspectral images corresponding to the Pavia University, Kennedy Space Center, and Botswana public hyperspectral datasets.…”

Variational Autoencoder With Optimizing Gaussian Mixture Model Priors

“…It can capture data distribution through neural networks and be used to generate new samples. Its variants have been widely studied and applied in various fields, such as unsupervised clustering [2] and image generation [3], [4]. VAE [1] belongs to the maximum likelihood generative model.…”

Topic Modeling on Triage Notes With Semiorthogonal Nonnegative Matrix Factorization

“…The enforcement of exact orthogonality also serves as a regularization by shrinking the model space. It eliminates multicollinearity between the basis vectors, and subsequently reduces the risk of overfitting (Abdi and Williams, 2010;Wang et al, 2019). This has advantages for both increasing the classification performance of subsequent supervised learning approaches by using the generated topic vectors as new features, and the interpretation of these topic vectors themselves.…”