Reconsidering Representation Alignment for Multi-view Clustering

Abstract

Self-supervised learning is a central component in recent approaches to deep multi-view clustering (MVC). However, we find large variations in the development of selfsupervision-based methods for deep MVC, potentially slowing the progress of the field. To address this, we present Deep-MVC, a unified framework for deep MVC that includes many recent methods as instances. We leverage our framework to make key observations about the effect of self-supervision, and in particular, drawbacks of aligning representations with contrastive learning. Further, we prove that contrastive alignment can negatively influence cluster separability, and that this effect becomes worse when the number of views increases. Motivated by our findings, we develop several new DeepMVC instances with new forms of self-supervision. We conduct extensive experiments and find that (i) in line with our theoretical findings, contrastive alignments decreases performance on datasets with many views; (ii) all methods benefit from some form of self-supervision; and (iii) our new instances outperform previous methods on several datasets. Based on our results, we suggest several promising directions for future research. To enhance the openness of the field, we provide an open-source implementation of Deep-MVC, including recent models and our new instances. Our implementation includes a consistent evaluation protocol, facilitating fair and accurate evaluation of methods and components 1 .* UiT Machine Learning group (machine-learning.uit.no) and Visual Intelligence Centre (visual-intelligence.no).† Norwegian Computing Center (nr.no).

This looks more like that: Enhancing Self-Explaining Models by Prototypical Relevance Propagation

“…Instead, deep-learning based multi-view clustering methodologies learn a common encoding with the help of deep neural networks, which can then be leveraged by the clustering module [51], [52]. The clustering module can, among others, be based on deep graph clustering [53], subspace clustering [54], adversarial based clustering methods [55], or contrastive learning [56].…”

“…We therefore include a few representative spectral multi-view clustering algorithms consisting of a two-step unweighted method [49], and a weighted method [50] both of which compute the Laplacian matrix and cluster assignments in two separate steps. Further, we compare results with a one-step method based on a rank constraint [48], which computes the similarities as well as cluster labels in one step, and with two recent deep learning based clustering methods [56].…”

“…The deep multi-view clustering in [56] first transforms each input (x v i , image i with view v) into its representation using view-specific encoders f v , as…”

“…This approach is termed as Simple Multi-View Clustering (SiMVC) by the authors, which, as the name suggests, is simple and efficient and works without explicit representation alignment. Further, [56] also introduce an auxiliary method which incorporates selective contrastive alignment of representations called Contrastive Multi-View Clustering (CoMVC).…”

“…Artifacts in the data can be learned by the model, which subsequently might lead to the model exhibiting undesirable behavior, as shown in recent works [1] and demonstrated above in case of the self-explaining network, ProtoPNet. Thus, it is essential to either remove the artifacts from the data, SpRAy-LRP [1] SpRAy-PRP [1] SiMVC [56] CoMVC [56] Co-Reg [49] WMSC [50] MCGC [ or to ensure that the model is not basing its decisions on these spurious attributes present in the data. We tried the latter in the experiments above via identifying and removing the artifact prototypes.…”