What Vision-Language Models `See' when they See Scenes

Cafagna, Michele; van Deemter, Kees; Gatt, Albert

doi:10.48550/arxiv.2109.07301

Cited by 4 publications

(4 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To the best of our knowledge, scene context has not yet been integrated into visual REG models (but see Cafagna et al 2021 for the inverse task of grounding descriptions in images). However, it is save to assume that leveraging this kind of context is a challenging modeling problem in itself.…”

Section: Toward a Wider Notion Of Context In Visual Regmentioning

confidence: 99%

Rethinking symbolic and visual context in Referring Expression Generation

Schüz¹,

Gatt²,

Zarrieß³

2023

Front. Artif. Intell.

Self Cite

View full text Add to dashboard Cite

Situational context is crucial for linguistic reference to visible objects, since the same description can refer unambiguously to an object in one context but be ambiguous or misleading in others. This also applies to Referring Expression Generation (REG), where the production of identifying descriptions is always dependent on a given context. Research in REG has long represented visual domains through symbolic information about objects and their properties, to determine identifying sets of target features during content determination. In recent years, research in visual REG has turned to neural modeling and recasted the REG task as an inherently multimodal problem, looking at more natural settings such as generating descriptions for objects in photographs. Characterizing the precise ways in which context influences generation is challenging in both paradigms, as context is notoriously lacking precise definitions and categorization. In multimodal settings, however, these problems are further exacerbated by the increased complexity and low-level representation of perceptual inputs. The main goal of this article is to provide a systematic review of the types and functions of visual context across various approaches to REG so far and to argue for integrating and extending different perspectives on visual context that currently co-exist in research on REG. By analyzing the ways in which symbolic REG integrates context in rule-based approaches, we derive a set of categories of contextual integration, including the distinction between positive and negative semantic forces exerted by context during reference generation. Using this as a framework, we show that so far existing work in visual REG has considered only some of the ways in which visual context can facilitate end-to-end reference generation. Connecting with preceding research in related areas, as possible directions for future research, we highlight some additional ways in which contextual integration can be incorporated into REG and other multimodal generation tasks.

show abstract

Section: Toward a Wider Notion Of Context In Visual Regmentioning

confidence: 99%

Rethinking symbolic and visual context in Referring Expression Generation

Schüz¹,

Gatt²,

Zarrieß³

2023

Front. Artif. Intell.

Self Cite

View full text Add to dashboard Cite

show abstract

“…For our evaluation, we choose CLIP, as opposed to other language-vision models, due to its vast popularity as a foundation model (Bommasani et al, 2022), i.e., its use in a multitude of models and its impressive zero-shot performance across various tasks and datasets, e.g., text-to-image retrieval, image question answering, human action segmentation, image-sentence alignment - (Cafagna et al, 2021). However, we observe these datasets contain mostly images from North America and Western Europe, and, to the best of our knowledge, we are the first to evaluate CLIP on more diverse data.…”

Section: State-of-the-art Vision-language Modelmentioning

confidence: 99%

Bridging the Digital Divide: Performance Variation across Socio-Economic Factors in Vision-Language Models

Nwatu,

Ignat,

Mihalcea

2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Despite the impressive performance of current AI models reported across various tasks, performance reports often do not include evaluations of how these models perform on the specific groups that will be impacted by these technologies. Among the minority groups underrepresented in AI, data from low-income households are often overlooked in data collection and model evaluation. We evaluate the performance of a state-of-the-art vision-language model (CLIP) on a geo-diverse dataset containing household images associated with different income values (Dollar Street) and show that performance inequality exists among households of different income levels. Our results indicate that performance for the poorer groups is consistently lower than the wealthier groups across various topics and countries. We highlight insights that can help mitigate these issues and propose actionable steps for economic-level inclusive AI development. Code is available at Analysis for Bridging the Digital Divide.

show abstract

“…Multimodal Grounding HL Dataset is also a useful resource to benchmark the grounding capabilities of large pre-trained V&L models. Along these lines, Cafagna et al (2021) study the capability of V&L models to understand scene descriptions in zero-shot settings, finding that only largescale pre-trained V&L models have enough generalization capabilities to handle unseen high-level scene descriptions. analyse the impact of exposure to high-level scene descriptions on multimodal representations in models pretrained on object-centric captions.…”

Section: Further Uses Of the Hl Datasetmentioning

confidence: 99%

HL Dataset: Visually-grounded Description of Scenes, Actions and Rationales

Cafagna,

van Deemter,

Gatt

2023

Proceedings of the 16th International Natural Language Generation Conference

View full text Add to dashboard Cite

Current captioning datasets focus on objectcentric captions, describing the visible objects in the image, e.g. "people eating food in a park". Although these datasets are useful to evaluate the ability of Vision & Language models to recognize and describe visual content, they do not support controlled experiments involving model testing or fine-tuning, with more high-level captions, which humans find easy and natural to produce. For example, people often describe images based on the type of scene they depict ('people at a holiday resort') and the actions they perform ('people having a picnic'). Such descriptions draw on personal experience and commonsense assumptions. We present the High-Level Dataset 1 ; a dataset extending 14997 images from the COCO dataset, aligned with a new set of 134,973 human annotated (high-level) captions collected along three axes: scenes, actions and rationales. We further extend this dataset with confidence scores collected from an independent set of readers, as well as a set of narrative captions generated synthetically, by combining each of the three axes. We describe this dataset and analyse it extensively. We also present baseline results for the High-Level Captioning task.

show abstract

What Vision-Language Models `See' when they See Scenes

Cited by 4 publications

References 32 publications

Rethinking symbolic and visual context in Referring Expression Generation

Rethinking symbolic and visual context in Referring Expression Generation

Bridging the Digital Divide: Performance Variation across Socio-Economic Factors in Vision-Language Models

HL Dataset: Visually-grounded Description of Scenes, Actions and Rationales

Contact Info

Product

Resources

About