CLIP-TD: CLIP Targeted Distillation for Vision-Language Tasks

Abstract

Contrastive language-image pretraining (CLIP) links vision and language modalities into a unified embedding space, yielding tremendous potential for vision-language (VL) tasks. While early concurrent works have begun to study this potential on a subset of tasks, important questions remain: 1) What is the benefit of CLIP on unstudied VL tasks? 2) Does CLIP provide benefit in low-shot or domain shifted scenarios? 3) Can CLIP improve existing approaches without impacting inference or pretraining complexity? In this work, we seek to answer these questions through two key contributions. First, we introduce an evaluation protocol that includes Visual Commonsense Reasoning (VCR), Visual Entailment (SNLI-VE), and Visual Question Answering (VQA), across a variety of data availability constraints and conditions of domain shift. Second, we propose an approach, named CLIP Targeted Distillation (CLIP-TD), to intelligently distill knowledge from CLIP into existing architectures using a dynamically weighted objective applied to adaptively selected tokens per instance. Experiments demonstrate that our proposed CLIP-TD leads to exceptional gains in the low-shot (up to 51.9%) and domain-shifted (up to 71.3%) conditions of VCR, while simultaneously improving performance under standard fullysupervised conditions (up to 2%), achieving state-of-art performance on VCR compared to other single models that are pretrained with image-text data only. On SNLI-VE, CLIP-TD produces significant gains in low-shot conditions (up to 6.6%) as well as fully supervised (up to 3%). On VQA, CLIP-TD provides improvement in low-shot (up to 9%), and in fully-supervised (up to 1.3%). Finally, CLIP-TD outperforms concurrent works utilizing CLIP for finetuning, as well as baseline naive distillation approaches. Code will be made available.

Open-Vocabulary Panoptic Segmentation with MaskCLIP

“…Many algorithms have been developed lately [34,31,36,24,26,28] trying knowledge distillation from the CLIP model to benefit the down-stream tasks one way or the other by leveraging the rich semantic language information paired in the images. Here, we directly adopt the backbone of CLIP image model to train for open-vocabulary panoptic/semantic segmentation.…”

Symmetrical Linguistic Feature Distillation with CLIP for Scene Text Recognition

“…It contains an image encoder and a text encoder to measure the content similarity of the given image and text. Many recent works have transferred it on multiple downstream tasks, including semantic segmentation [8,24], object detection [5], Visual Question Answering [48] and image generation [33]. Many researchers regard CLIP as a pre-trained feature extractor [8,24,26,38,52].…”

Dataset Bias Mitigation in Multiple-Choice Visual Question Answering and Beyond

“…With UM and DS biases appearing to be dominant in text information, existing VL models are encouraged to learn shortcuts in text and under-utilize the visual information leading to false visual dependency, as in Figure 6a (Cao et al, 2020;Wang et al, 2022c;Dancette et al, 2021;Chen et al, 2020a). To assist models in mitigating UM and DS biases, we design ADS-I to synthesize positive images I+ and negative images Ito assist models' training and emphasize the correct question-related visual dependency.…”