Deep Robust Clustering by Contrastive Learning

Abstract

Recently, many unsupervised deep learning methods have been proposed to learn clustering with unlabelled data. By introducing data augmentation, most of the latest methods look into deep clustering from the perspective that the original image and its transformation should share similar semantic clustering assignment. However, the representation features could be quite different even they are assigned to the same cluster since softmax function is only sensitive to the maximum value. This may result in high intra-class diversities in the representation feature space, which will lead to unstable local optimal and thus harm the clustering performance. To address this drawback, we proposed Deep Robust Clustering (DRC). Different from existing methods, DRC looks into deep clustering from two perspectives of both semantic clustering assignment and representation feature, which can increase inter-class diversities and decrease intra-class diversities simultaneously. Furthermore, we summarized a general framework that can turn any maximizing mutual information into minimizing contrastive loss by investigating the internal relationship between mutual information and contrastive learning. And we successfully applied it in DRC to learn invariant features and robust clusters. Extensive experiments on six widely-adopted deep clustering benchmarks demonstrate the superiority of DRC in both stability and accuracy. e.g., attaining 71.6% mean accuracy on CIFAR-10, which is 7.1% higher than state-of-the-art results.

Graph Contrastive Clustering

“…According the difference in self-supervised signal, deep clustering methods can be mainly divided into two categories, including the reconstruction based methods [37,28,8,11,38] and the self-augmentation based methods [3,35,17,12,16,33,42].…”

“…PICA [16] learns the most semantically plausible clustering solution by maximizing partition confidence. DRC [42] tries to learn invariant features and clusters by introducing contrastive learning to optimize the consistency between image and its augmentation simultaneously. SCAN [33] utilizes a three-stage method to improve the clustering.…”

“…Recently, contrastive learning [4] has received much attention in unsupervised feature learning, which emphasizes the importance of data augmentation and maximizes the agreement between two augmented samples. Because of its success, a few approaches [16,42,23,35] are proposed to jointly optimize the contrastive learning and clustering. For instance, partition confidence maximisation (PICA) [16] learns the most semantically plausible clustering solution by maximizing partition confidence, which corresponds to the cluster-wise contrastive learning.…”

“…For instance, partition confidence maximisation (PICA) [16] learns the most semantically plausible clustering solution by maximizing partition confidence, which corresponds to the cluster-wise contrastive learning. Instead of only using the cluster contrast in PICA, deep robust clustering (DRC) [42] adopts the conventional contrastive learning in feature and cluster space simultaneously. These methods significantly improve the clustering performance, but they still face another obvious issue: both of them still follow the basic framework of contrastive learning and only assume that a sample and its augmentations should be similar in the feature space, which does not incorporate the latent category information into clustering.…”

Multi-view Contrastive Graph Clustering

“…Several loss functions have been proposed, such as the triplet loss [Chopra et al, 2005], the noise contrastive estimation (NCE) loss [Gutmann and Hyvärinen, 2010], the normalized temperature-scaled cross entropy loss (NT-Xent) . Deep robust clustering turns maximizing mutual information into minimizing contrastive loss and achieves significant improvement after applying contrastive learning to decrease intra-class variance [Zhong et al, 2020]. Contrastive clustering develops a dual contrastive learning framework, which conducts contrastive learning at instance-level as well as cluster-level [Li et al, 2021].…”