Investigating Negation in Pre-trained Vision-and-language Models

Dobreva, Radina; Keller, Frank

doi:10.18653/v1/2021.blackboxnlp-1.27

Cited by 5 publications

(6 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A series of studies (van Miltenburg et al, 2016(van Miltenburg et al, , 2017, investigated negation in Flickr30k image descriptions using a smaller set of negation words compared to our study, comparing the use of negation in English, German, and Dutch, and finding no significant differences. Dobreva and Keller (2021) show that the performance of vision and language models decreases when the text contains negation, but did not show that this decrease is caused by negation-related visual features. Text-only models also have difficulty processing negations (e.g., Ettinger (2020)), and the drop in performance could be due to the text encoder alone.…”

Section: Computational Studiesmentioning

confidence: 79%

See 1 more Smart Citation

A Large-Scale Multilingual Study of Visual Constraints on Linguistic Selection of Descriptions

Berger¹,

Frermann²,

Stanovsky³

et al. 2023

Preprint

View full text Add to dashboard Cite

We present a large, multilingual study into how vision constrains linguistic choice, covering four languages and five linguistic properties, such as verb transitivity or use of numerals. We propose a novel method that leverages existing corpora of images with captions written by native speakers, and apply it to nine corpora, comprising 600k images and 3M captions. We study the relation between visual input and linguistic choices by training classifiers to predict the probability of expressing a property from raw images, and find evidence supporting the claim that linguistic properties are constrained by visual context across languages. We complement this investigation with a corpus study, taking the test case of numerals. Specifically, we use existing annotations (number or type of objects) to investigate the effect of different visual conditions on the use of numeral expressions in captions, and show that similar patterns emerge across languages. Our methods and findings both confirm and extend existing research in the cognitive literature. We additionally discuss possible applications for language generation. We make our codebase publicly available. 1

show abstract

Section: Computational Studiesmentioning

confidence: 79%

“…Negation words (Neg). We use the list of English negation words composed by Dobreva and Keller (2021), and add the word nope. We translate all words in the English list into the other languages, and verify the resulting lists with a native speaker.…”

Section: We Use Microsoft'smentioning

confidence: 99%

A Large-Scale Multilingual Study of Visual Constraints on Linguistic Selection of Descriptions

Berger¹,

Frermann²,

Stanovsky³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Human-level Intelligence Numerous studies have been conducted to examine the linguistic capabilities of pre-trained transformer-based language models, exploring their resemblance to human abilities in various aspects such as syntactic knowledge (Linzen et al, 2016;Gulordava et al, 2018), semantic knowledge (Ettinger, 2020;Kementchedjhieva et al, 2021;Misra et al, 2020), and the integration of semantic and syntactic information (Xu and Chen, 2022). However, there is still limited knowledge about the linguistic capabilities of VLP models in relation to human behavior (Dobreva and Keller, 2021;Cao et al, 2020a). Our study seeks to expand the current understanding by investigating whether the impact of tags with various features on the VQA task is comparable to the effect of distractors in the PWI task.…”

Section: Related Workmentioning

confidence: 99%

“…Numerous studies exploring human-like behavior in models primarily focus on language processing alone, revealing that, even though these models may encounter difficulties in certain specialized areas of language, they can attain significant human-like capabilities across diverse linguistic domains (Ettinger, 2020;Rogers et al, 2021). In this study, we expand the scope of the investigation to the field of vision and language, an area that, to the best of our knowledge, has been relatively unexplored from the perspective of the language community (Dobreva and Keller, 2021;Cao et al, 2020a).…”

Section: Introductionmentioning

confidence: 99%

Human-Like Distractor Response in Vision-Language Model

Xu,

Chen

2023

Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacifi

View full text Add to dashboard Cite

Previous studies exploring the human-like capabilities of machine-learning models have primarily focused on pure language models. Limited attention has been given to investigating whether models exhibit human-like behavior when performing tasks that require the integration of visual and language information. In this study, we investigate the impact of tags of semantic, phonological, and bilingual features on the visual question-answering task performance of an unsupervised model. Our findings reveal its similarities with the influence of distractors in the picture-naming task (known as the picture-word-interference paradigm) observed in human experiments: 1) Semanticallyrelated tags have a more negative effect on task performance compared to unrelated tags, indicating a more robust competition between visual and tag information which are semantically closer to each other when generating an answer. 2) Even presenting a partial section (wordpiece) of the originally detected tag significantly improves task performance, with the portion that plays a lesser role in determining the overall meaning of the original tag leading to a more pronounced improvement. 3) Tags in two languages that refer to the same meaning exhibit a symmetrical-like effect on performance in balanced bilingual models. Datasets and code of this project are released at https: //github.com/NLPbelllabs/PWI

show abstract

“…Leveraging a FOIL-like paradigm, subsequent studies revealed that Transformer-based models struggle when dealing with verb arguments (SVO-Probes; Hendricks and Nematzadeh, 2021), negation (Dobreva and Keller, 2021), numbers, spatial relations (VALSE; , and expressions requiring compositional abilities (WinoGround;Thrush et al, 2022;Diwan et al, 2022) or embedding semantically underspecified language (Pezzelle, 2023). Crucially, all this previous work focused on phenomena and tasks that require more than a basic language understanding to be properly mastered.…”

Section: Language Abilities Of Pre-trained Multimodal Modelsmentioning

confidence: 99%

The BLA Benchmark: Investigating Basic Language Abilities of Pre-Trained Multimodal Models

Chen,

Fernández,

Pezzelle

2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Despite the impressive performance achieved by pre-trained language-and-vision models in downstream tasks, it remains an open question whether this reflects a proper understanding of image-text interaction. In this work, we explore to what extent they handle basic linguistic constructions-active-passive voice, coordination, and relative clauses-that even preschool children can typically master. We present BLA, a novel, automatically constructed benchmark to evaluate multimodal models on these Basic Language Abilities. We show that different types of Transformer-based systems, such as CLIP, ViLBERT, and BLIP2, generally struggle with BLA in a zero-shot setting, in line with previous findings. Our experiments, in particular, show that most of the tested models only marginally benefit when fine-tuned or prompted with construction-specific samples. Yet, the generative BLIP2 shows promising trends, especially in an in-context learning setting. This opens the door to using BLA not only as an evaluation benchmark but also to improve models' basic language abilities.

show abstract

Investigating Negation in Pre-trained Vision-and-language Models

Cited by 5 publications

References 17 publications

A Large-Scale Multilingual Study of Visual Constraints on Linguistic Selection of Descriptions

A Large-Scale Multilingual Study of Visual Constraints on Linguistic Selection of Descriptions

Human-Like Distractor Response in Vision-Language Model

The BLA Benchmark: Investigating Basic Language Abilities of Pre-Trained Multimodal Models

Contact Info

Product

Resources

About