What Does BERT with Vision Look At?

Li, Liunian Harold; Yatskar, Mark; Yin, Dong; Hsieh, Cho‐Jui; Chang, Kai-Wei

doi:10.18653/v1/2020.acl-main.469

Cited by 151 publications

(177 citation statements)

References 28 publications

Supporting

Mentioning

177

Contrasting

Order By: Relevance

“…Devlin et al (2019) first proposed to pre-train a large Transformer architecture, called BERT, to learn representations of natural language using large-scale unlabeled data in a self-supervised fashion. Later, BERT's task-independent pre-training approach is rigorously studied (Devlin et al, 2019;Solaiman et al, 2019;Feng et al, 2020;Li et al, 2020). While BERT-like models have shown effectiveness in learning contextualized representation, it is not very useful in generation tasks.…”

Section: Related Workmentioning

confidence: 99%

Unified Pre-training for Program Understanding and Generation

Ahmad¹,

Chakraborty²,

Ray³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

Self Cite

341

146

View full text Add to dashboard Cite

Code summarization and generation empower conversion between programming language (PL) and natural language (NL), while code translation avails the migration of legacy code from one PL to another. This paper introduces PLBART, a sequence-to-sequence model capable of performing a broad spectrum of program and language understanding and generation tasks. PLBART is pre-trained on an extensive collection of Java and Python functions and associated NL text via denoising autoencoding. Experiments on code summarization in the English language, code generation, and code translation in seven programming languages show that PLBART outperforms or rivals state-of-the-art models. Moreover, experiments on discriminative tasks, e.g., program repair, clone detection, and vulnerable code detection, demonstrate PLBART's effectiveness in program understanding. Furthermore, analysis reveals that PLBART learns program syntax, style (e.g., identifier naming convention), logical flow (e.g., if block inside an else block is equivalent to else if block) that are crucial to program semantics and thus excels even with limited annotations.

show abstract

Section: Related Workmentioning

confidence: 99%

Unified Pre-training for Program Understanding and Generation

Ahmad¹,

Chakraborty²,

Ray³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

Self Cite

341

146

View full text Add to dashboard Cite

show abstract

“…paraphrases) and align visual with linguistic elements. Li et al (2020) hinted the importance of attention-based vision-and-language model's ability to map entity-words to corresponding image regions. Following this direction and to improve a model's reasoning abilities, we propose to further fine-tune a pre-trained model with the aim of learning visually grounded paraphrases (VGPs) (Chu et al, 2018;Otani et al, 2020).…”

Section: Vgp Fine-tuningmentioning

confidence: 99%

“…On the other hand, the internal behaviors of vision-and-language models have attracted less interest from the research community. Li et al (2020) have shown some attention heads in vision-andlanguage models are able to map entities to image regions while others even detect syntactic relations between non-entity words and image regions. Nevertheless, no initiative has been taken towards supervising directly the attention modules.…”

Section: Introductionmentioning

confidence: 99%

Attending Self-Attention: A Case Study of Visually Grounded Supervision in Vision-and-Language Transformers

Samaran¹,

García

Otani

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

The impressive performances of pre-trained visually grounded language models have motivated a growing body of research investigating what has been learned during the pre-training. As a lot of these models are based on Transformers, several studies on the attention mechanisms used by the models to learn to associate phrases with their visual grounding in the image have been conducted. In this work, we investigate how supervising attention directly to learn visual grounding can affect the behavior of such models. We compare three different methods on attention supervision and their impact on the performances of a state-of-the-art visually grounded language model on two popular vision-and-language tasks.

show abstract

“…Current state-of-the-art image captioning models use a pre-trained object detector to generate features and spatial information of the objects present in an image (e.g., as in Oscar [12]). In particular, [12] utilizes BERT-like objectives to learn cross-modal representation on different vision-language tasks (similar ideas form the basis of recent pre-trained multi-modal models, e.g., VisualBERT [11]). Prior captioning approaches have involved attention mechanisms and their variants to capture spatial relationship between objects [6] for generating captions.…”

Section: Textual Description Of Images: Object Detection and Image Captioningmentioning

confidence: 99%

VisualTextRank: Unsupervised Graph-based Content Extraction for Automating Ad Text to Image Search

Mishra

Kuznetsov

Srivastava

et al. 2021

Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery &Amp; Data Mining

View full text Add to dashboard Cite

Numerous online stock image libraries offer high quality yet copyright free images for use in marketing campaigns. To assist advertisers in navigating such third party libraries, we study the problem of automatically fetching relevant ad images given the ad text (via a short textual query for images). Motivated by our observations in logged data on ad image search queries (given ad text), we formulate a keyword extraction problem, where a keyword extracted from the ad text (or its augmented version) serves as the ad image query. In this context, we propose VisualTextRank: an unsupervised method to (i) augment input ad text using semantically similar ads, and (ii) extract the image query from the augmented ad text. Visu-alTextRank builds on prior work on graph based context extraction (biased TextRank in particular) by leveraging both the text and image of similar ads for better keyword extraction, and using advertiser category specific biasing with sentence-BERT embeddings. Using data collected from the Verizon Media Native (Yahoo Gemini) ad platform's stock image search feature for onboarding advertisers, we demonstrate the superiority of VisualTextRank compared to competitive keyword extraction baselines (including an 11% accuracy lift over biased TextRank). For the case when the stock image library is restricted to English queries, we show the effectiveness of VisualTextRank on multilingual ads (translated to English) while leveraging semantically similar English ads. Online tests with a simplified version of VisualTextRank led to a 28.7% increase in the usage of stock image search, and a 41.6% increase in the advertiser onboarding rate in the Verizon Media Native ad platform. CCS CONCEPTS• Information systems → Online advertising.

show abstract

What Does BERT with Vision Look At?

Cited by 151 publications

References 28 publications

Unified Pre-training for Program Understanding and Generation

Unified Pre-training for Program Understanding and Generation

Attending Self-Attention: A Case Study of Visually Grounded Supervision in Vision-and-Language Transformers

VisualTextRank: Unsupervised Graph-based Content Extraction for Automating Ad Text to Image Search

Contact Info

Product

Resources

About