Data Efficient Language-supervised Zero-shot Recognition with Optimal Transport Distillation

Wu, Bichen; Cheng, Ruizhe; Zhang, Peizhao; Vajda, P.

doi:10.48550/arxiv.2112.09445

Cited by 4 publications

(4 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We follow zero-shot CLIP benchmark 7 implementation for most of the datasets, and implement the ones that are missing. For most image classification tasks we compute Accuracy@1, except HatefulMemes where we compute AUROC because it is binary classification, OpenImages where we compute FlatHit@1 following [75], and PascalVOC2007 where we compute mean average precision (mAP) because it is multi-label classification. We use the same prompt ensembling method as CLIP [61] to improve zero-shot image classification.…”

Section: A3 Evaluation Detailsmentioning

confidence: 99%

Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Training

Radenović¹,

Dubey²,

Kadian³

et al. 2023

Preprint

View full text Add to dashboard Cite

Vision-language models trained with contrastive learning on large-scale noisy data are becoming increasingly popular for zero-shot recognition problems. In this paper we improve the following three aspects of the contrastive pre-training pipeline: dataset noise, model initialization and the training objective. First, we propose a straightforward filtering strategy titled Complexity, Action, and Textspotting (CAT) that significantly reduces dataset size, while achieving improved performance across zero-shot visionlanguage tasks. Next, we propose an approach titled Concept Distillation to leverage strong unimodal representations for contrastive training that does not increase training complexity while outperforming prior work. Finally, we modify the traditional contrastive alignment objective, and propose an importance-sampling approach to up-sample the importance of hard-negatives without adding additional complexity. On an extensive zero-shot benchmark of 29 tasks, our Distilled and Hard-negative Training (DiHT) approach improves on 20 tasks compared to the baseline. Furthermore, for few-shot linear probing, we propose a novel approach that bridges the gap between zero-shot and fewshot performance, substantially improving over prior work. Models are available at github.com/facebookresearch/diht.

show abstract

Section: A3 Evaluation Detailsmentioning

confidence: 99%

Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Training

Radenović¹,

Dubey²,

Kadian³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…The core technique of CLIP is aligning both vision and language modalities in a joint embedding space by global representation contrastive. Followup works further improve CLIP from the vision-only [43,67] or vision-language [38,64,68,69] side. In this paper, we bring natural language supervision together with masked image modeling for better visual pre-training on these two paradigms.…”

Section: Related Workmentioning

confidence: 99%

RILS: Masked Visual Reconstruction in Language Semantic Space

Yang¹,

Ge²,

Yi³

et al. 2023

Preprint

View full text Add to dashboard Cite

Both masked image modeling (MIM) and natural language supervision have facilitated the progress of transferable visual pre-training. In this work, we seek the synergy between two paradigms and study the emerging properties when MIM meets natural language supervision. To this end, we present a novel masked visual Reconstruction In Language semantic Space (RILS) pre-training framework, in which sentence representations, encoded by the text encoder, serve as prototypes to transform the vision-only signals into patch-sentence probabilities as semantically meaningful MIM reconstruction targets. The vision models can therefore capture useful components with structured information by predicting proper semantic of masked tokens. Better visual representations could, in turn, improve the text encoder via the image-text alignment objective, which is essential for the effective MIM target transformation. Extensive experimental results demonstrate that our method not only enjoys the best of previous MIM and CLIP but also achieves further improvements on various tasks due to their mutual benefits. RILS exhibits advanced transferability on downstream classification, detection, and segmentation, especially for low-shot regimes. Code will be made available at https://github.com/hustvl/RILS.

show abstract

“…The definition of OT has been applied to many areas, such as domain adaptation [11], generative models [6,33] and self-supervision learning [24,40]. For tasks involving cross-lingual settings, Nguyen and Luu [29] employed OT distance as a part of the loss function in a knowledge distillation framework for improving the cross-lingual summarization.…”

Section: Optimal Transportmentioning

confidence: 99%

Improving Cross-lingual Information Retrieval on Low-Resource Languages via Optimal Transport Distillation

Huang,

Yu,

Allan

2023

Preprint

View full text Add to dashboard Cite

Benefiting from transformer-based pre-trained language models, neural ranking models have made significant progress. More recently, the advent of multilingual pre-trained language models provides great support for designing neural cross-lingual retrieval models. However, due to unbalanced pre-training data in different languages, multilingual language models have already shown a performance gap between high and low-resource languages in many downstream tasks. And cross-lingual retrieval models built on such pre-trained models can inherit language bias, leading to suboptimal result for low-resource languages. Moreover, unlike the English-to-English retrieval task, where large-scale training collections for document ranking such as MS MARCO are available, the lack of cross-lingual retrieval data for low-resource language makes it more challenging for training cross-lingual retrieval models. In this work, we propose OPTICAL: Optimal Transport distillation for low-resource Cross-lingual information retrieval. To transfer a model from high to low resource languages, OPTICAL forms the cross-lingual token alignment task as an optimal transport problem to learn from a well-trained monolingual retrieval model. By separating the cross-lingual knowledge from knowledge of query document matching, OPTICAL only needs bitext data for distillation training, which is more feasible for low-resource languages. Experimental results show that, with minimal training data, OPTI-CAL significantly outperforms strong baselines on low-resource languages, including neural machine translation. CCS CONCEPTS• Information systems → Information retrieval; Multilingual and cross-lingual retrieval; Retrieval models and ranking.

show abstract

Data Efficient Language-supervised Zero-shot Recognition with Optimal Transport Distillation

Cited by 4 publications

References 35 publications

Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Training

Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Training

RILS: Masked Visual Reconstruction in Language Semantic Space

Improving Cross-lingual Information Retrieval on Low-Resource Languages via Optimal Transport Distillation

Contact Info

Product

Resources

About