Multimodal Adaptive Distillation for Leveraging Unimodal Encoders for Vision-Language Tasks

Wang, Zhecan; Codella, Noel; Chen, Yen-Chun; Zhou, Luowei; Dai, Xiyang; Xiao, Bin; Yang, Jun; You, Haoxuan; Chang, Kai-Wei; Chang, Shih-Fu; Liu, Yuan

doi:10.48550/arxiv.2204.10496

Cited by 3 publications

(4 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As their models sizes grow rapidly, recent works have also explored parameter-efficient learning and model compression methods, including adapters (Sung, Cho, and Bansal 2022;Houlsby et al 2019;Rebuffi, Bilen, andVedaldi 2018, 2017), prompt tuning (Gu et al 2021b (Wang et al 2020b;Li et al 2022c;Wang et al 2021;Ding et al 2023), distilling VL models is a relatively under-explored field, as pointed out by the review paper of (Chen et al 2022a). (Fang et al 2021) claim to be the first to distill vision-language Transformers, and MAD (Wang et al 2022b) claims to be the first to use multi-modal distillation for VL models. These two papers both focus on distilling Encoder-only VL Transformers.…”

Section: Related Workmentioning

confidence: 99%

No Head Left Behind – Multi-Head Alignment Distillation for Transformers

Zhao,

Singh,

Appalaraju

et al. 2024

AAAI

View full text Add to dashboard Cite

Knowledge distillation aims at reducing model size without compromising much performance. Recent work has applied it to large vision-language (VL) Transformers, and has shown that attention maps in the multi-head attention modules of vision-language Transformers contain extensive intra-modal and cross-modal co-reference relations to be distilled. The standard approach is to apply a one-to-one attention map distillation loss, i.e. the Teacher's first attention head instructs the Student's first head, the second teaches the second, and so forth, but this only works when the numbers of attention heads in the Teacher and Student are the same. To remove this constraint, we propose a new Attention Map Alignment Distillation (AMAD) method for Transformers with multi-head attention, which works for a Teacher and a Student with different numbers of attention heads. Specifically, we soft-align different heads in Teacher and Student attention maps using a cosine similarity weighting. The Teacher head contributes more to the Student heads for which it has a higher similarity weight. Each Teacher head contributes to all the Student heads by minimizing the divergence between the attention activation distributions for the soft-aligned heads. No head is left behind. This distillation approach operates like cross-attention. We experiment on distilling VL-T5 and BLIP, and apply AMAD loss on their T5, BERT, and ViT sub-modules. We show, under vision-language setting, that AMAD outperforms conventional distillation methods on VQA-2.0, COCO captioning, and Multi30K translation datasets. We further show that even without VL pre-training, the distilled VL-T5 models outperform corresponding VL pre-trained VL-T5 models that are further fine-tuned by ground-truth signals, and that fine-tuning distillation can also compensate to some degree for the absence of VL pre-training for BLIP models.

show abstract

Section: Related Workmentioning

confidence: 99%

No Head Left Behind – Multi-Head Alignment Distillation for Transformers

Zhao,

Singh,

Appalaraju

et al. 2024

AAAI

View full text Add to dashboard Cite

show abstract

“…This is exemplified by models directly learning the shallow mapping between prior question words and shared class labels in the absence of sample-specific contextualized candidate options. Consequently, models develop false visual dependency (Cao et al, 2020;Wang et al, 2022b) as they may succeed in resolving VQA tasks (Selvaraju et al, 2016;Chen et al, 2020a;Gupta et al, 2022) utilizing irrelevant visual cues.…”

Section: Related Workmentioning

confidence: 99%

“…For instance, training strategies like (Gupta et al, 2022; and DS solutions, (Ray et al, 2019;Selvaraju et al, 2020;Ribeiro et al, 2019;Wang et al, 2022d) only focus on a single modality. Debiased training like (Wang et al, 2022b;Zhang et al, 2021b) require constraints of either a specific model structure or doubling the models' complexity. Other methods (Chen et al, 2020a;Liang et al, 2020) apply occlusion boxes or maskings on images or questions and thus drastically disturb data distribution, leading to nonsensical synthesized answers.…”

Section: Related Workmentioning

confidence: 99%

Dataset Bias Mitigation in Multiple-Choice Visual Question Answering and Beyond

Wang,

Chen,

You

et al. 2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

Vision-language (VL) understanding tasks evaluate models' comprehension of complex visual scenes through multiple-choice questions. However, we have identified two dataset biases that models can exploit as shortcuts to resolve various VL tasks correctly without proper understanding. The first type of dataset bias is Unbalanced Matching bias, where the correct answer overlaps the question and image more than the incorrect answers. The second type of dataset bias is Distractor Similarity bias, where incorrect answers are overly dissimilar to the correct answer but significantly similar to other incorrect answers within the same sample. To address these dataset biases, we first propose Adversarial Data Synthesis (ADS) to generate synthetic training and debiased evaluation data. We then introduce Intra-sample Counterfactual Training (ICT) to assist models in utilizing the synthesized training data, particularly the counterfactual data, via focusing on intra-sample differentiation. Extensive experiments demonstrate the effectiveness of ADS and ICT in consistently improving model performance across different benchmarks, even in domain-shifted scenarios.

show abstract

“…Previous works (Song et al, 2022;Subramanian et al, 2022;Shen et al, 2021;Wang et al, 2022b) demonstrated that CLIP can achieve strong zeroshot performance of vision-language tasks by converting original tasks into the image-text matching format. However, they mainly consider matching on an instance or global level, i.e., the whole image or sentence, ignoring the significance of finegrained elements, e.g., keywords in the sentence and objects in the image.…”

Section: Introductionmentioning

confidence: 99%

UniFine: A Unified and Fine-grained Approach for Zero-shot Vision-Language Understanding

Sun¹,

Wang²,

You³

et al. 2023

Findings of the Association for Computational Linguistics: ACL 2023

View full text Add to dashboard Cite

Vision-language tasks, such as VQA, SNLI-VE, and VCR are challenging because they require the model's reasoning ability to understand the semantics of the visual world and natural language. Supervised methods working for vision-language tasks have been wellstudied. However, solving these tasks in a zero-shot setting is less explored. Since Contrastive Language-Image Pre-training (CLIP) has shown remarkable zero-shot performance on image-text matching, previous works utilized its strong zero-shot ability by converting vision-language tasks into an image-text matching problem, and they mainly consider global-level matching (e.g., the whole image or sentence). However, we find visual and textual fine-grained information, e.g., keywords in the sentence and objects in the image, can be fairly informative for semantics understanding. Inspired by this, we propose a unified framework to take advantage of the fine-grained information for zero-shot visionlanguage learning, covering multiple tasks such as VQA, SNLI-VE, and VCR. Our experiments show that our framework outperforms former zero-shot methods on VQA and achieves substantial improvement on SNLI-VE and VCR. Furthermore, our ablation studies confirm the effectiveness and generalizability of our proposed method. Code is available at https://github.com/ThreeSR/UniFine.

show abstract

Multimodal Adaptive Distillation for Leveraging Unimodal Encoders for Vision-Language Tasks

Cited by 3 publications

References 0 publications

No Head Left Behind – Multi-Head Alignment Distillation for Transformers

No Head Left Behind – Multi-Head Alignment Distillation for Transformers

Dataset Bias Mitigation in Multiple-Choice Visual Question Answering and Beyond

UniFine: A Unified and Fine-grained Approach for Zero-shot Vision-Language Understanding

Contact Info

Product

Resources

About