Learning Representation for Clustering Via Prototype Scattering and Positive Sampling

Abstract

Existing deep clustering methods rely on either contrastive or non-contrastive representation learning for downstream clustering task. Contrastive-based methods thanks to negative pairs learn uniform representations for clustering, in which negative pairs, however, may inevitably lead to the class collision issue and consequently compromise the clustering performance. Non-contrastive-based methods, on the other hand, avoid class collision issue, but the resulting non-uniform representations may cause the collapse of clustering. To enjoy the strengths of both worlds, this paper presents a novel end-to-end deep clustering method with prototype scattering and positive sampling, termed ProPos. Specifically, we first maximize the distance between prototypical representations, named prototype scattering loss, which improves the uniformity of representations. Second, we align one augmented view of instance with the sampled neighbors of another view-assumed to be truly positive pair in the embedding space-to improve the within-cluster compactness, termed positive sampling alignment. The strengths of ProPos are avoidable class collision issue, uniform representations, well-separated clusters, and within-cluster compactness. By optimizing ProPos in an end-to-end expectation-maximization framework, extensive experimental results demonstrate that ProPos achieves competing performance on moderate-scale clustering benchmark datasets and establishes new state-of-the-art performance on large-scale datasets. Source code is available at https://github.com/Hzzone/ProPos.

Deep Multiview Clustering via Iteratively Self-Supervised Universal and Specific Space Learning

“…Based on the assumption that neighboring samples around one view are truly positive with respect to its augmented view [12], we propose diffused sampling to spread the current features to the domain to generate diffused features. The key goal of DSA is to generate the diffused features.…”

“…The key goal of DSA is to generate the diffused features. Specifically, following [12], we achieve the generation of diffused features by introducing a Gaussian distribution, which can be expressed as follows,…”

“…Similar to vanilla contrastive learning, PCL treats one prototype (i.e. cluster center) and the prototype of another view as positive pairs, while considering other prototypes as negative pairs [12]. Specifically, assume that there are K prototypes from the student embedding space, {u s 1 , u s 2 , .…”

“…Unlike some clustering algorithms [6], [9] and knowledge distillation methods [17], [18] requiring an additional pre-training stage or a well-trained teacher model, our HaDis is pretraining-free, as its teacher network gradually updates along with the student network. Furthermore, to avoid collapse and unstable clustering [12], we introduce prototypical contrastive learning (PCL) [11], [12] into our framework, which conducts contrastive learning at the prototype-level, leading to enhancing both the efficiency of clustering and the quality of clustering results.…”

Transfer Learning in Motor Imagery Brain Computer Interface: A Review

“…It minimizes the distance between feature representations to learn richer information from the teacher model compared to softened labels [49]. Further, contrastive representation distillation is proposed to exploit the structural characteristics among different samples by harnessing the discriminative representations, thereby enhancing differentiation within data [8,19,45]. This work distills knowledge from the intermediate-view images to enhance the sparseview CT reconstruction.…”