Can contrastive learning avoid shortcut solutions?

Robinson, Joshua A.; Li, Sun; Yu, Ke; Batmanghelich, Kayhan; Jegelka, Stefanie; Sra, Suvrit

doi:10.48550/arxiv.2106.11230

Cited by 5 publications

(10 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It provides a principled way to finding more discriminative negatives for an effective training of the encoder. Along this line, an implicit feature modification (IFM) [17] is proposed to iteratively update the hard samples. The IFM also pushes the samples towards the representation of anchors like the AdCo, but its step size is set by a given level of budget.…”

Section: Related Workmentioning

confidence: 99%

CaCo: Both Positive and Negative Samples are Directly Learnable via Cooperative-adversarial Contrastive Learning

Wang¹,

Huang²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

As a representative self-supervised method, contrastive learning has achieved great successes in unsupervised training of representations. It trains an encoder by distinguishing positive samples from negative ones given query anchors. These positive and negative samples play critical roles in defining the objective to learn the discriminative encoder, avoiding it from learning trivial features. While existing methods heuristically choose these samples, we present a principled method where both positive and negative samples are directly learnable end-to-end with the encoder. We show that the positive and negative samples can be cooperatively and adversarially learned by minimizing and maximizing the contrastive loss, respectively. This yields cooperative positives and adversarial negatives with respect to the encoder, which are updated to continuously track the learned representation of the query anchors over mini-batches. The proposed method achieves 71.3% and 75.3% in top-1 accuracy respectively over 200 and 800 epochs of pre-training ResNet-50 backbone on ImageNet1K without tricks such as multi-crop or stronger augmentations. With Multi-Crop, it can be further boosted into 75.7%. The source code and pre-trained model are released in https://github.com/maple-research-lab/caco.

show abstract

Section: Related Workmentioning

confidence: 99%

CaCo: Both Positive and Negative Samples are Directly Learnable via Cooperative-adversarial Contrastive Learning

Wang¹,

Huang²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…However, neither of these approaches is considered in the image retrieval setup. [50] propose adversarial training in the latent space with the goal of improving standard generalization. Closer to image attribution, Panum et al [47] combine deep metric learning algorithms with adversarial training but they perform only small-scale experiments and evaluate their models via nearest neighbour classification which is different from the retrieval under editorial and non-editorial distortions as we do in the context of image attribution.…”

Section: Related Workmentioning

confidence: 99%

ARIA: Adversarially Robust Image Attribution for Content Provenance

Andriushchenko¹,

Li²,

Oxholm³

et al. 2022

Preprint

View full text Add to dashboard Cite

Image attribution -matching an image back to a trusted source -is an emerging tool in the fight against online misinformation. Deep visual fingerprinting models have recently been explored for this purpose. However, they are not robust to tiny input perturbations known as adversarial examples. First we illustrate how to generate valid adversarial images that can easily cause incorrect image attribution. Then we describe an approach to prevent imperceptible adversarial attacks on deep visual fingerprinting models, via robust contrastive learning. The proposed training procedure leverages training on ∞ -bounded adversarial examples, it is conceptually simple and incurs only a small computational overhead. The resulting models are substantially more robust, are accurate even on unperturbed images, and perform well even over a database with millions of images. In particular, we achieve 91.6% standard and 85.1% adversarial recall under ∞ -bounded perturbations on manipulated images compared to 80.1% and 0.0% from prior work. We also show that robustness generalizes to other types of imperceptible perturbations unseen during training. Finally, we show how to train an adversarially robust image comparator model for detecting editorial changes in matched images.

show abstract

“…How well self-supervised representations generalize to different settings, after training, is often assessed using a down-stream evaluation task, such as object classification [6] or speaker identification [40] Shortcut feature representations. Robinson et al [49] show that the features learned by the InfoNCE [40] loss depend on the difficulty of instance discrimination during training. If the instance discrimination task is easy to solve during training, the model will learn shortcut features.…”

Section: Generalizability Of Contrastive Lossesmentioning

confidence: 99%

“…How well an ICR method generalizes beyond the specific training setup depends on the features that the method has learned during training. Contrastive loss functions are prone to learning shortcuts [30,49], which are rules that perform well on standard benchmarks but fail to generalize to other testing conditions [13]. In the context of ICR, a shortcut is a latent representation of either the image or the caption that does not contain all the aspects mentioned in a caption that describes that scene.…”

Section: Introductionmentioning

confidence: 99%

“…If a caption describes certain aspects of a scene that are ignored by either the image or the caption encoder, these aspects cannot be used for retrieval (i.e., the encoder has learned a shortcut to match certain images and captions). Features extracted during training depend on the difficulty of the discrimination task, i.e., how difficult it is for the model to discriminate positive and negative candidates [49]. Shortcut feature representations are problematic from a generalization perspective: if certain features are not learned during training, even though this information is present in the training data, this information cannot be used during evaluation (i.e., for retrieval).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Keep the Caption Information: Preventing Shortcut Learning in Contrastive Image-Caption Retrieval

Bleeker¹,

Yates²,

Rijke³

2022

Preprint

View full text Add to dashboard Cite

To train image-caption retrieval (ICR) methods, contrastive loss functions are a common choice for optimization functions. Unfortunately, contrastive ICR methods are vulnerable to learning shortcuts: decision rules that perform well on the training data but fail to transfer to other testing conditions. We introduce an approach to reduce shortcut feature representations for the ICR task: latent target decoding (LTD). We add an additional decoder to the learning framework to reconstruct the input caption, which prevents the image and caption encoder from learning shortcut features. Instead of reconstructing input captions in the input space, we decode the semantics of the caption in a latent space. We implement the LTD objective as an optimization constraint, to ensure that the reconstruction loss is below a threshold value while primarily optimizing for the contrastive loss. Importantly, LTD does not depend on additional training data or expensive (hard) negative mining strategies. Our experiments show that, unlike reconstructing the input caption, LTD reduces shortcut learning and improves generalizability by obtaining higher recall@k and r-precision scores. Additionally, we show that the evaluation scores benefit from implementing LTD as an optimization constraint instead of a dual loss.

show abstract

Can contrastive learning avoid shortcut solutions?

Cited by 5 publications

References 11 publications

CaCo: Both Positive and Negative Samples are Directly Learnable via Cooperative-adversarial Contrastive Learning

CaCo: Both Positive and Negative Samples are Directly Learnable via Cooperative-adversarial Contrastive Learning

ARIA: Adversarially Robust Image Attribution for Content Provenance

Keep the Caption Information: Preventing Shortcut Learning in Contrastive Image-Caption Retrieval

Contact Info

Product

Resources

About