Noise Is Also Useful: Negative Correlation-Steered Latent Contrastive Learning

Yan, Jiexi; Luo, Lei; Xu, Chenghao; Deng, Cheng; Huang, Heng

doi:10.1109/cvpr52688.2022.00013

Cited by 15 publications

(4 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, the work (Jian, Gao, and Vosoughi 2022) contrasts examples from text and examples from another modality simultaneously while learning sentence embeddings. In (Yan et al 2022), a contrastive learning method is exploited in the latent metric space to explore the useful negative correlation hidden in noisy data, which can improve the robustness of DNNs. Similarly, the work (Wang et al 2021) extends contrastive learning to the multilabel classification task.…”

Section: Contrastive Learningmentioning

confidence: 99%

A Learnable Discrete-Prior Fusion Autoencoder with Contrastive Learning for Tabular Data Synthesis

Zhang,

Lou,

et al. 2024

AAAI

View full text Add to dashboard Cite

The actual collection of tabular data for sharing involves confidentiality and privacy constraints, leaving the potential risks of machine learning for interventional data analysis unsafely averted. Synthetic data has emerged recently as a privacy-protecting solution to address this challenge. However, existing approaches regard discrete and continuous modal features as separate entities, thus falling short in properly capturing their inherent correlations. In this paper, we propose a novel contrastive learning guided Gaussian Transformer autoencoder, termed GTCoder, to synthesize photo-realistic multimodal tabular data for scientific research. Our approach introduces a transformer-based fusion module that seamlessly integrates multimodal features, permitting for mining more informative latent representations. The attention within the fusion module directs the integrated output features to focus on critical components that facilitate the task of generating latent embeddings. Moreover, we formulate a contrastive learning strategy to implicitly constrain the embeddings from discrete features in the latent feature space by encouraging the similar discrete feature distributions closer while pushing the dissimilar further away, in order to better enhance the representation of the latent embedding. Experimental results indicate that GTCoder is effective to generate photo-realistic synthetic data, with interactive interpretation of latent embedding, and performs favorably against some baselines on most real-world and simulated datasets.

show abstract

Section: Contrastive Learningmentioning

confidence: 99%

A Learnable Discrete-Prior Fusion Autoencoder with Contrastive Learning for Tabular Data Synthesis

Zhang,

Lou,

et al. 2024

AAAI

View full text Add to dashboard Cite

show abstract

“…Then we dig into the details of the normal distribution of the policy initialization models. In the Adaptive Sampling with Reward (ASR) setting (Dou et al 2022b), the policy initialization distribution guides a sampling distribution of p (O n | O a ), which builds on the distance between negative sample O n and anchor sample O a in triplet loss construction (Chen et al 2019;Yan et al 2022;Dou, Luo, and Yang 2022). A high variance normal distribution policy initialization means that at the beginning of the sampling stage, the model will sample negative samples from a long range of distances from the anchor sample, which includes hard, semi-hard, and easy negative samples.…”

Section: Main Contributionsmentioning

confidence: 99%

Demystify the Gravity Well in the Optimization Landscape (Student Abstract)

Dou

Bao

Song

et al. 2023

AAAI

View full text Add to dashboard Cite

We provide both empirical and theoretical insights to demystify the gravity well phenomenon in the optimization landscape. We start from describe the problem setup and theoretical results (an escape time lower bound) of the Softmax Gravity Well (SGW) in the literature. Then we move toward the understanding of a recent observation called ASR gravity well. We provide an explanation of why normal distribution with high variance can lead to suboptimal plateaus from an energy function point of view. We also contribute to the empirical insights of curriculum learning by comparison of policy initialization by different normal distributions. Furthermore, we provide the ASR escape time lower bound to understand the ASR gravity well theoretically. Future work includes more specific modeling of the reward as a function of time and quantitative evaluation of normal distribution’s influence on policy initialization.

show abstract

“…Zhang et al (2021) proposes to sample counterfactual instances and learn inter-modality relationships by contrastive learning for visual commonsense reasoning. Contrastive learning is also widely used for weakly supervised learning (Du et al, 2022;Yan et al, 2022;Gao et al, 2022;Xie et al, 2022;Li et al, 2022b;Wan et al, 2022;Yi et al, 2022;Chen et al, 2021). In this paper, we leverage contrastive learning to address the bias and the noise problems in DS-MRC.…”

Section: Relate Workmentioning

confidence: 99%

Contrastive Distant Supervision for Debiased and Denoised Machine Reading Comprehension

Bian,

Lin,

Han

et al. 2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

Distant Supervision (DS) is a promising learning approach for MRC by leveraging easilyobtained question-answer pairs. Unfortunately, the heuristically annotated dataset will inevitably lead to mislabeled instances, resulting in answer bias and context noise problems. To learn debiased and denoised MRC models, this paper proposes the Contrastive Distant Supervision algorithm -CDS, which can learn to distinguish confusing and noisy instances via confidence-aware contrastive learning. Specifically, to eliminate answer bias, CDS samples counterfactual negative instances, which ensures that MRC models must take both answer information and question-context interaction into consideration. To denoise distantly annotated contexts, CDS samples confusing negative instances to increase the margin between correct and mislabeled instances. We further propose a confidence-aware contrastive loss to model and leverage the uncertainty of all DS instances during learning. Experimental results show that CDS is effective and can even outperform supervised MRC models without manual annotations.

show abstract

Noise Is Also Useful: Negative Correlation-Steered Latent Contrastive Learning

Cited by 15 publications

References 21 publications

A Learnable Discrete-Prior Fusion Autoencoder with Contrastive Learning for Tabular Data Synthesis

A Learnable Discrete-Prior Fusion Autoencoder with Contrastive Learning for Tabular Data Synthesis

Demystify the Gravity Well in the Optimization Landscape (Student Abstract)

Contrastive Distant Supervision for Debiased and Denoised Machine Reading Comprehension

Contact Info

Product

Resources

About