Large-Scale Adversarial Training for Vision-and-Language Representation Learning

Abstract

We present VILLA, the first known effort on large-scale adversarial training for vision-and-language (V+L) representation learning. VILLA consists of two training stages: (i) task-agnostic adversarial pre-training; followed by (ii) task-specific adversarial finetuning. Instead of adding adversarial perturbations on image pixels and textual tokens, we propose to perform adversarial training in the embedding space of each modality. To enable large-scale training, we adopt the "free" adversarial training strategy, and combine it with KL-divergence-based regularization to promote higher invariance in the embedding space. We apply VILLA to current best-performing V+L models, and achieve new state of the art on a wide range of tasks, including Visual Question Answering, Visual Commonsense Reasoning, Image-Text Retrieval, Referring Expression Comprehension, Visual Entailment, and NLVR 2 .We propose VILLA (Vision-and-Language Large-scale Adversarial training), which advocates the use of adversarial training for V+L representation learning. As illustrated in Figure 1, VILLA consists of two training stages: (i) task-agnostic adversarial pre-training (APT); followed by (ii) task-specific adversarial fine-tuning (AFT). Intuitively, if well-designed, multimodal pre-training tasks such as image-conditioned masked language modeling and image-text matching can resonate well with many downstream tasks that require visual grounding and reasoning abilities. This leads to our hypothesis that the improved generalization ability of pre-trained models learned during APT stage can be readily transferred to the AFT stage for diverse tasks. In other words, APT is able to uniformly lift model performance for all downstream tasks in a task-agnostic way, while AFT can further enhance the finetuned models by leveraging task-specific supervision signals.To bring in more flexibility in generating adversarial examples for robust training, we propose to perform adversarial training on the embedding level for multi-modalities, instead of operating on Preprint. Under review.

Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey

“…Similarly, [48] and [284] It is also claimed by Salman et al [286] that adversarially trained models, while less accurate than the standard models, often perform better for transfer learning. In another study, Gan et al [287] propose VILLA, a representation learning approach based on large-scale adversarial training on visionand-language data. They perform a task-agnostic adversarial training followed by a task-specific adversarial fine-tuning in the embedding space.…”

SimVLM: Simple Visual Language Model Pretraining with Weak Supervision

“…Recent years have seen a rapid progress made in vision-language pretraining (Uppal et al, 2020;Han et al, 2021;Khan et al, 2021). While a variety of approaches have been proposed, a large portion of them require object detection for image region feature regression or tagging as part of the pre-training objectives, for example LXMERT (Tan & Bansal, 2019), VLBERT (Su et al, 2020), VisualBERT (Li et al, 2019), UNITER (Chen et al, 2020b), Villa (Gan et al, 2020), Oscar , ERNIE-ViL (Yu et al, 2021), UNIMO , VinVL , VIVO VL-T5 (Cho et al, 2021) etc. These methods rely on a strong object detection model like Fast(er) R-CNN (Ren et al, 2015), which is often trained on human annotated data sets like Visual Genome (Krishna et al, 2016).…”

“…To examine the quality of vision-language pretraining, we first compare SimVLM on the popular multi-modal tasks described in Sec. 4.1.2 with state-of-the-art (SOTA) VLP methods including LXMERT (Tan & Bansal, 2019), VL-T5 (Cho et al, 2021), UNITER (Chen et al, 2020b), OSCAR , Villa (Gan et al, 2020), SOHO , UNIMO , and VinVL .…”

A Thousand Words Are Worth More Than a Picture: Natural Language-Centric Outside-Knowledge Visual Question Answering

“…Multiple VQA datasets have been proposed, such as Visual Genome QA [25] VQA [2], GQA [16], CLEVR [22], MovieQA [53] and so on. Many works have shown state-of-the-art performance on VQA tasks, including task-specific VQA models with various cross-modality fusion mechanisms [13,20,24,49,62,66,67] and joint vision-language models that are pretrained on large-scale vision-language corpus and finetuned on VQA tasks [6,11,29,30,33,52,68]. Please note that the conventional VQA task does not require external knowledge by definition, although studies show some VQA questions may require commonsense knowledge to answer correctly [2].…”