A Closer Look at the Robustness of Vision-and-Language Pre-trained Models

Li, Linjie; Gan, Zhe; Liu, Jingjing

doi:10.48550/arxiv.2012.08673

Cited by 13 publications

(15 citation statements)

References 59 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Robust VQA Benchmarks. Following [32], we also evaluate our model over on a suite of robust VQA benchmarks. VQA-Rephrasings [48] exposes VQA models to linguistic variations in questions, and measures consistency of model predicitions to different semantivally equivalent questions.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

MLP Architectures for Vision-and-Language Modeling: An Empirical Study

Nie¹,

Fu²,

Gan³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

We initiate the first empirical study on the use of MLP architectures for vision-and-language (VL) fusion. Through extensive experiments on 5 VL tasks and 5 robust VQA benchmarks, we find that: (i) Without pre-training, using MLPs for multimodal fusion has a noticeable performance gap compared to transformers; (ii) However, VL pre-training can help close the performance gap; (iii) Instead of heavy multi-head attention, adding tiny one-head attention to MLPs is sufficient to achieve comparable performance to transformers. Moreover, we also find that the performance gap between MLPs and transformers is not widened when being evaluated on the "harder" robust VQA benchmarks, suggesting using MLPs for VL fusion can generalize roughly to a similar degree as using transformers. These results hint that MLPs can effectively learn to align vision and text features extracted from lower-level encoders without heavy reliance on self-attention. Based on this, we ask an even bolder question: can we have an all-MLP architecture for VL modeling, where both VL fusion and the vision encoder are replaced with MLPs? Our result shows that an all-MLP VL model is sub-optimal compared to stateof-the-art full-featured VL models when both of them get pre-trained. However, pre-training an all-MLP can surprisingly achieve a better average score than full-featured transformer models without pre-training. This indicates the potential of large-scale pre-training of MLP-like architectures for VL modeling and inspires the future research direction on simplifying well-established VL modeling with less inductive design bias. Our code is publicly available at: https://github.com/easonnie/mlp-vil.

show abstract

Section: Methodsmentioning

confidence: 99%

“…We adopt the same strategy as VQA and use binary cross entropy to optimize the model training. For datasets built on the original VQAv2 val splits, we follow [32] to finetune all models on VQAv2 training split to avoid data contamination. For adversarial VQA datasets, we follow [33,50] to evaluate models finetuned on VQAv2+VG-QA.…”

Section: Appendixmentioning

confidence: 99%

MLP Architectures for Vision-and-Language Modeling: An Empirical Study

Nie¹,

Fu²,

Gan³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Along the journey of VLP, researchers have investigated different training strategies [12,35], robustness [29], compression [10,11,47], probing analysis [3,31], and the extension to video-text modeling [24,28,43,44,59]. More recently, instead of using object detectors for image feature extraction, end-to-end VLP based on convolution networks and transformers are becoming popular [17,18,22,27,49].…”

Section: Related Workmentioning

confidence: 99%

Scaling Up Vision-Language Pre-training for Image Captioning

Hu¹,

Gan²,

Wang³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

In recent years, we have witnessed significant performance boost in the image captioning task based on visionlanguage pre-training (VLP). Scale is believed to be an important factor for this advance. However, most existing work only focuses on pre-training transformers with moderate sizes (e.g., 12 or 24 layers) on roughly 4 million images. In this paper, we present LEMON , a LargE-scale iMage captiONer, and provide the first empirical study on the scaling behavior of VLP for image captioning. We use the state-of-the-art VinVL model as our reference model, which consists of an image feature extractor and a transformer model, and scale the transformer both up and down, with model sizes ranging from 13 to 675 million parameters. In terms of data, we conduct experiments with up to 200 million image-text pairs which are automatically collected from web based on the alt attribute of the image (dubbed as ALT200M). Extensive analysis helps to characterize the performance trend as the model size and the pre-training data size increase. We also compare different training recipes, especially for training on large-scale noisy data. As a result, LEMON achieves new state of the arts on several major image captioning benchmarks, including COCO Caption, nocaps, and Conceptual Captions. We also show LEMON can generate captions with long-tail visual concepts when used in a zero-shot manner.

show abstract

“…Visual-linguistic Pre-training. Following the prominent progress in the transformer-based [63] pretraining in natural language [13,47,32,4,10,48], visual-linguistic pre-training models, either for im-age+text [39,59,8,37,20,73,36,15,35,40] or for video+text [56,35,41,75,33], have achieved great success on a number of downstream V+L tasks. Most existing VL models are designed in a two-step fashion: a pre-trained object detector is used to encode the image as set of regional features (as offline visual tokens) followed by pre-training on a large scale visual-linguistic corpus using tasks like masked language modeling, image-text matching or masked region modeling losses.…”

Section: Related Workmentioning

confidence: 99%

Compressing Visual-linguistic Model via Knowledge Distillation

Fang

Wang

et al. 2021

Preprint

View full text Add to dashboard Cite

Despite exciting progress in pre-training for visuallinguistic (VL) representations, very few aspire to a small VL model. In this paper, we study knowledge distillation (KD) to effectively compress a transformer based large VL model into a small VL model. The major challenge arises from the inconsistent regional visual tokens extracted from different detectors of Teacher and Student, resulting in the misalignment of hidden representations and attention distributions. To address the problem, we retrain and adapt the Teacher by using the same region proposals from Student's detector while the features are from Teacher's own object detector. With aligned network inputs, the adapted Teacher is capable of transferring the knowledge through the intermediate representations. Specifically, we use the mean square error loss to mimic the attention distribution inside the transformer block, and present a token-wise noise contrastive loss to align the hidden state by contrasting with negative representations stored in a sample queue. To this end, we show that our proposed distillation significantly improves the performance of small VL models on image captioning and visual question answering tasks. It reaches 120.8 in CIDEr score on COCO captioning, an improvement of 5.1 over its non-distilled counterpart; and an accuracy of 69.8 on VQA 2.0, a 0.8 gain from the baseline. Our extensive experiments and ablations confirm the effectiveness of VL distillation in both pre-training and fine-tuning stages.

show abstract

A Closer Look at the Robustness of Vision-and-Language Pre-trained Models

Cited by 13 publications

References 59 publications

MLP Architectures for Vision-and-Language Modeling: An Empirical Study

MLP Architectures for Vision-and-Language Modeling: An Empirical Study

Scaling Up Vision-Language Pre-training for Image Captioning

Compressing Visual-linguistic Model via Knowledge Distillation

Contact Info

Product

Resources

About