VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomena

Pârcălăbescu, Letiția; Cafagna, Michele; Muradjan, Lilitta; Frank, Anette; Calixto, Iacer; Gatt, Albert

doi:10.48550/arxiv.2112.07566

Cited by 6 publications

(14 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The second strand of work in curating adversarial samples are efforts revolving around red-teaming (Ganguli et al, 2022;Perez et al, 2022) that aim to explicitly elicit certain sets of behaviours from foundation models; primarily these approaches look at the problem of adversarial benchmarking from a safety perspective. Further, a host of benchmarks that aim to stress-test models are making their way on the horizon-their primary goal is to create test sets for manually discovered failure modes (Yuksekgonul et al, 2022;Parcalabescu et al, 2021;Thrush et al, 2022;Udandarao et al, 2023;Hsieh et al, 2023;Kamath et al, 2023;Bitton-Guetta et al, 2023;Bordes et al, 2023). However, while they are sample efficient, they are criticized as unfair.…”

Section: Extended Related Workmentioning

confidence: 99%

GDumb: A Simple Approach that Questions Our Progress in Continual Learning

Prabhu

Torr

Dokania

2020

Lecture Notes in Computer Science

345

283

View full text Add to dashboard Cite

Standardized benchmarks drive progress in machine learning. However, with repeated testing, the risk of overfitting grows as algorithms overexploit benchmark idiosyncrasies. In our work, we seek to mitigate this challenge by compiling ever-expanding large-scale benchmarks called Lifelong Benchmarks. As exemplars of our approach, we create Lifelong-CIFAR10 and Lifelong-ImageNet, containing (for now) 1.69M and 1.98M test samples, respectively. While reducing overfitting, lifelong benchmarks introduce a key challenge: the high cost of evaluating a growing number of models across an ever-expanding sample set. To address this challenge, we also introduce an efficient evaluation framework: Sort & Search (S&S), which reuses previously evaluated models by leveraging dynamic programming algorithms to selectively rank and sub-select test samples, enabling cost-effective lifelong benchmarking. Extensive empirical evaluations across ∼31,000 models demonstrate that S&S achieves highly-efficient approximate accuracy measurement, reducing compute cost from 180 GPU days to 5 GPU hours (∼1000x reduction) on a single A100 GPU, with low approximation error. As such, lifelong benchmarks offer a robust, practical solution to the "benchmark exhaustion" problem.

show abstract

Section: Extended Related Workmentioning

confidence: 99%

GDumb: A Simple Approach that Questions Our Progress in Continual Learning

Prabhu

Torr

Dokania

2020

Lecture Notes in Computer Science

345

283

View full text Add to dashboard Cite

show abstract

“…The Visual Commonsense Tests (ViComTe) dataset (Zhang et al, 2022a) is created to test to what degree unimodal (language-only) and multimodal (image and language) models capture a broad range of visually salient attributes. VALSE (Parcalabescu et al, 2021) is proposed to test VLP models centered on linguistic phenomena. CARET (Jimenez et al, 2022) is proposed to systematically measure consistency and robustness of modern VQA models through six fine-grained capability tests.…”

Section: Robustness and Probing Analysismentioning

confidence: 99%

Vision-Language Pre-training: Basics, Recent Advances, and Future Trends

Gan¹,

Fu²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

This paper surveys vision-language pre-training (VLP) methods for multimodal intelligence that have been developed in the last few years. We group these approaches into three categories: (i) VLP for image-text tasks, such as image captioning, image-text retrieval, visual question answering, and visual grounding; (ii) VLP for core computer vision tasks, such as (open-set) image classification, object detection, and segmentation; and (iii) VLP for video-text tasks, such as video captioning, video-text retrieval, and video question answering. For each category, we present a comprehensive review of state-of-the-art methods, and discuss the progress that has been made and challenges still being faced, using specific systems and models as case studies. In addition, for each category, we discuss advanced topics being actively explored in the research community, such as big foundation models, unified modeling, in-context few-shot learning, knowledge, robustness, and computer vision in the wild, to name a few.♠ Zhe Gan and Jianfeng Gao initiated the project. Zhe Gan and Linjie Li took lead in the writing of Chapter 1. Linjie Li and Jianfeng Gao took lead in the writing of Chapter 2. Zhe Gan further took lead in the writing of Chapter 3 and 7. Chunyuan Li took lead in the writing of Chapter 4. Linjie Li further took lead in the writing of Chapter 5. Lijuan Wang and Zicheng Liu took lead in the writing of Chapter 6. All the authors provided project advice, and contributed to paper editing and proofreading.

show abstract

“…Visual Language Inference With the advent of visual language models (VLMs; Liu et al 2021;Li et al 2019;Cho et al 2021;Huang et al 2022) that can simultaneously process visual and linguistic information, there is growing attention to enrich text-only tasks with visual context (Parcalabescu et al, 2021;Xie et al, 2018;Vu et al, 2018). Vu et al (2018) propose a visuallygrounded version of the textual entailment task, supported by the cognitive science view of enriching meaning representations with multiple modalities.…”

Section: Related Workmentioning

confidence: 99%

“…Second, we propose three strategies for retrieving a rich amount of cheap and allowably noisy supervision signals for inference and rationalization. Similar to Parcalabescu et al (2021), PVLIR's three strategies rely on the available image captioning datasets (e.g. Changpinyo et al (2021); Sharma et al (2018); Gurari et al (2020a); Lin et al (2014a)) that are readily available as a result of years of research in the field and maturity of resources.…”

Section: Introductionmentioning

confidence: 99%

Factors affecting students’ speaking performance in English department of Jawzjan University

Qasemi

2020

ijirss

View full text Add to dashboard Cite

This study aims to recognize the problems that students of English department face when they speak English. To achieve the objective of the investigation a quantitative research was designed. A questionnaire was used as the main data collection tool. The findings of the study showed that majority of the students have a positive perception about importance and necessity of speaking skill. According to the findings most of the students do not have the opportunity of practicing English speaking out of the class. Moreover, dissimilarity in English education background of the students are another challenge which is recognized in outcome of the study. Furthermore, the findings showed that most of the students feel motivated and confident in speaking performance. Moreover, the findings also revealed that the teachers have significant role for motivating the students in speaking performance. To sum up, the result exposed that students are appreciated and praised when they perform a speaking task efficiently.

show abstract

VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomena

Cited by 6 publications

References 33 publications

GDumb: A Simple Approach that Questions Our Progress in Continual Learning

GDumb: A Simple Approach that Questions Our Progress in Continual Learning

Vision-Language Pre-training: Basics, Recent Advances, and Future Trends

Factors affecting students’ speaking performance in English department of Jawzjan University

Contact Info

Product

Resources

About