Partially View-aligned Representation Learning with Noise-robust Contrastive Loss

Yang, Mouxing; Li, Yunfan; Huang, Zhenyu; Liu, Zitao; Hu, Peng; Peng, Xi

doi:10.1109/cvpr46437.2021.00119

Cited by 118 publications

(36 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Yuan et al [48] propose a method combined intra-modal and inter-modal similarity preservation objectives to improve the quality of learned representation. To alleviate the influence of false negatives caused by random sampling, Yang et al [45] propose a noise-robust contrastive loss to simultaneously learn representation and align data in multi-view learning. The existing works heavily rely on the correct correspondence between modalities.…”

Section: Multi-modal Contrastive Learningmentioning

confidence: 99%

Noise-Tolerant Learning for Audio-Visual Action Recognition

Han¹,

Luo²,

Miao³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recently, video recognition is emerging with the help of multi-modal learning, which focuses on integrating multiple modalities to improve the performance or robustness of a model. Although various multi-modal learning methods have been proposed and offer remarkable recognition results, almost all of these methods rely on high-quality manual annotations and assume that modalities among multi-modal data provide relevant semantic information. Unfortunately, most widely used video datasets are collected from the Internet and inevitably contain noisy labels and noisy correspondence. To solve this problem, we use the audio-visual action recognition task as a proxy and propose a noise-tolerant learning framework to find anti-interference model parameters to both noisy labels and noisy correspondence. Our method consists of two phases and aims to rectify noise by the inherent correlation between modalities. A noise-tolerant contrastive training phase is performed first to learn robust model parameters unaffected by the noisy labels. To reduce the influence of noisy correspondence, we propose a cross-modal noise estimation component to adjust the consistency between different modalities. Since the noisy correspondence existed at the instance level, a category-level contrastive loss is proposed to further alleviate the interference of noisy correspondence. Then in the hybrid supervised training phase, we calculate the distance metric among features to obtain corrected labels, which are used as complementary supervision. In addition, we investigate the noisy correspondence in real-world datasets and conduct comprehensive experiments with synthetic and real noise data. The results verify the advantageous performance of our method compared to state-of-the-art methods.

show abstract

Section: Multi-modal Contrastive Learningmentioning

confidence: 99%

Noise-Tolerant Learning for Audio-Visual Action Recognition

Han¹,

Luo²,

Miao³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…, 𝝅, and Lagrangian multipliers by solving (11), ( 12), ( 15), ( 19), ( 24), ( 29), ( 36), ( 37), (40), and ( 43 check convergence condition:…”

Section: Algorithm 1 Mvsc-cslfmentioning

confidence: 99%

Partially latent factors based multi‐view subspace learning

Liu

et al. 2022

Computational Intelligence

View full text Add to dashboard Cite

Multi-view subspace clustering always performs well in high-dimensional data analysis, but is sensitive to the quality of data representation. To this end, a two-stage fusion strategy is proposed to embed representation learning into the process of multi-view subspace clustering. This article first proposes a novel matrix factorization method that can separate the coupling consistent and complementary information from observations of multiple views. Based on the obtained latent representations, we further propose two subspace clustering strategies: feature-level fusion and subspace-level hierarchical strategy. The feature-level method concatenates all kinds of latent representations from multiple views, and the original problem therefore degenerates to a single-view subspace clustering process. The subspace-level hierarchical method performs different self-expressive reconstruction processes on the corresponding complementary and consistent latent representations coming from each view, that is, the prior constraints imposed on different types of subspace representations are related to the relevant input factors. Finally, extensive experimental results on real-world datasets demonstrate the superiority of our proposed methods by Note that the shorter conference version of this article was published in the International Joint Conference on Neural Networks (2021). In the conference version, we only proposed the subspace-level hierarchical method MVSC-CSLFS, and another feature-level fusion algorithm MVSC-CSLF is proposed in this manuscript. Besides, we analyze and describe the proposed algorithms and validate the performances on multiple datasets.

show abstract

“…In order to discover clusters from partial multi-modal data, various partial multi-modal (partial multi-modal (PMC)) methods have emerged [31][32][33]. The PMC method aimed to build a shared latent subspace of a complete modality to compensate for the latent representation of missing data.…”

Section: Partial Multi-modal Clusteringmentioning

confidence: 99%

Two‐stage partial image‐text clustering (TPIT‐C)

Guo

Lian

et al. 2022

IET Computer Vision

View full text Add to dashboard Cite

Deep multi-model clustering is a challenging task for data analysis since it learns a universal semantic representation to find correct clusters from heterogeneous samples. However, most existing methods 1) lack an effective approach to getting a global representation of visual instances, which results in a huge semantic gap between visual and textual space. 2) hardly consider partial multi-modal, where each instance is represented by only one modality. In reality, the pairing information for modalities is not available for all instances. To tackle the above issues, we propose a novel model called the Two-Stage Partial Image-Text Clustering (TPIT-C) model. Firstly, we build an interpretable reasoning network to obtain the salient regions and semantic concepts of the scene in order to generate global semantic concepts. Secondly, we construct an adversarial learning module to align textual and visual instances into a unified space by virtue of cycle-consistency. The experimental evaluations on public unpaired multi-model datasets illustrated that the proposed method has better performance and the effectiveness of our algorithm in the partial image-text clustering task.This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited.

show abstract

Partially View-aligned Representation Learning with Noise-robust Contrastive Loss

Cited by 118 publications

References 20 publications

Noise-Tolerant Learning for Audio-Visual Action Recognition

Noise-Tolerant Learning for Audio-Visual Action Recognition

Partially latent factors based multi‐view subspace learning

Two‐stage partial image‐text clustering (TPIT‐C)

Contact Info

Product

Resources

About