Probing Cross-Modal Representations in Multi-Step Relational Reasoning

Parfenova, Iuliia; Elliott, Desmond; Fernández, Raquel; Pezzelle, Sandro

doi:10.18653/v1/2021.repl4nlp-1.16

Cited by 2 publications

(2 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moreover, our study can inform computational work on question generation in the domain of natural language processing (see, e.g., Wang and Lake, 2021 ). Asking (informative) questions could also be of crucial importance, for example, to multimodal AI models asked to provide a correct answer to a question regarding the content of an image (Antol et al., 2015 ; Johnson et al., 2017 ) or the abstract relation tying various scenes depicting similar objects (Parfenova, Elliott, Fernández, & Pezzelle, 2021 ). This possibility could reduce the uncertainty of a model—when the input question is vague, ambiguous, or can be misinterpreted—and drive its decisions toward the correct output.…”

Section: Discussionmentioning

confidence: 99%

Semantic Adaptation to the Interpretation of Gradable Adjectives via Active Linguistic Interaction

Pezzelle

Fernández

2023

Cognitive Science

Self Cite

View full text Add to dashboard Cite

When communicating, people adapt their linguistic representations to those of their interlocutors. Previous studies have shown that this also occurs at the semantic level for vague and context-dependent terms such as quantifiers and uncertainty expressions. However, work to date has mostly focused on passive exposure to a given speaker's interpretation, without considering the possible role of active linguistic interaction. In this study, we focus on gradable adjectives big and small and develop a novel experimental paradigm that allows participants to ask clarification questions to figure out their interlocutor's interpretation. We find that, when in doubt, speakers do resort to this strategy, despite its inherent cognitive cost, and that doing so results in higher semantic alignment measured in terms of communicative success. While not all question-answer pairs are equally informative, we show that speakers become better questioners as the interaction progresses. Yet, the higher semantic alignment observed when speakers are able to ask questions does not increase over time. This suggests that conversational interaction's key advantage may be to boost coordination without committing to long-term semantic updates. Our findings shed new light on the mechanisms used by speakers to achieve semantic alignment and on how language is shaped by communication.

show abstract

Section: Discussionmentioning

confidence: 99%

Semantic Adaptation to the Interpretation of Gradable Adjectives via Active Linguistic Interaction

Pezzelle

Fernández

2023

Cognitive Science

Self Cite

View full text Add to dashboard Cite

show abstract

“…Finally, Parfenova et al (2021) recently created a dataset of three-image scenes to probe two-step reasoning.…”

Section: Related Workmentioning

confidence: 99%

QLEVR: A Diagnostic Dataset for Quantificational Language and Elementary Visual Reasoning

Li¹,

Søgaard²

2022

Preprint

View full text Add to dashboard Cite

Synthetic datasets have successfully been usedto probe visual question-answering datasets for their reasoning abilities. CLEVR (Johnson et al., 2017), for example, tests a range of visual reasoning abilities. The questions in CLEVR focus on comparisons of shapes, colors, and sizes, numerical reasoning, and existence claims. This paper introduces a minimally biased, diagnostic visual questionanswering dataset, QLEVR, that goes beyond existential and numerical quantification and focus on more complex quantifiers and their combinations, e.g., asking whether there are more than two red balls that are smaller than at least three blue balls in an image. We describe how the dataset was created and present a first evaluation of state-of-the-art visual question-answering models, showing that QLEVR presents a formidable challenge to our current models. Code and Dataset are available at https://github.com/ zechenli03/QLEVR

show abstract

Probing Cross-Modal Representations in Multi-Step Relational Reasoning

Cited by 2 publications

References 29 publications

Semantic Adaptation to the Interpretation of Gradable Adjectives via Active Linguistic Interaction

Semantic Adaptation to the Interpretation of Gradable Adjectives via Active Linguistic Interaction

QLEVR: A Diagnostic Dataset for Quantificational Language and Elementary Visual Reasoning

Contact Info

Product

Resources

About