Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality

Abstract

We present a novel task and dataset for evaluating the ability of vision and language models to conduct visio-linguistic compositional reasoning, which we call Winoground. Given two images and two captions, the goal is to match them correctly-but crucially, both captions contain a completely identical set of words, only in a different order. The dataset was carefully hand-curated by expert annotators and is labeled with a rich set of fine-grained tags to assist in analyzing model performance. We probe a diverse range of state-of-the-art vision and language models and find that, surprisingly, none of them do much better than chance. Evidently, these models are not as skilled at visio-linguistic compositional reasoning as we might have hoped. We perform an extensive analysis to obtain insights into how future work might try to mitigate these models' shortcomings. We aim for Winoground to serve as a useful evaluation set for advancing the state of the art and driving further progress in the field. The dataset is available at https://huggingface.co/datasets/facebook/winoground.

How to Adapt Pre-trained Vision-and-Language Models to a Text-only Input?

“…In contrast to their work, we investigate several VL models and evaluate their performance on language generation tasks. Bugliarello et al (2021), Hessel and Lee (2020), Thrush et al (2022) and Frank et al (2021) also perform extensive evaluations of several VL models such as LXMERT and VisualBERT. In contrast to our work, they primarily focus on the VL performance of the models, and do not consider the model performance on text-only input.…”

Research in information systems at the University of Maryland

“…Image-text models that have been constrastively trained on internet-scale data, such as CLIP (Radford et al 2021a), have been shown to have strong zero-shot classification capabilities. However, recent works (Thrush et al 2022;Diwan et al 2022) have highlighted their limitations in visio-linguistic reasoning, as shown in the challenging Winoground benchmark. Yuksekgonul et al (2023) also observe this issue and introduce a new benchmark ARO for image-text models which require a significant amount of visio-linguistic reasoning to solve.…”

“…Winoground (Thrush et al 2022;Diwan et al 2022) is a challenging vision-language dataset for evaluating the visiolinguistic characteristics of contrastively trained image-text models. The dataset consists of 400 tasks, where each task consists of two image-text pairs.…”

“…In the past few years, image-text contrastively pre-trained multimodal models such as CLIP (Radford et al 2021a) have shown tremendous ability to perform zero-shot classification (Mu et al 2021;Minderer et al 2022), imagetext retrieval (Diwan et al 2022;Thrush et al 2022) and image-captioning (Yu et al 2022;Li et al 2022;Mokady, Hertz, and Bermano 2021). These contrastive models are also used as a part of various state-of-the-art pipelines for downstream tasks such as segmentation (Wang et al 2021;Lüddecke and Ecker 2021), object-detection (Minderer et al 2022;Zhong et al 2021) and model interpretability (Moayeri et al 2023).…”

“…However, recent works have shown that these models fail on visio-linguistic reasoning tasks, for example identifying the relative position between objects in an image. In fact, the performance of CLIP on Winoground (Thrush et al 2022;Diwan et al 2022), a challenging benchmark for * These authors contributed equally.…”