Grounded Textual Entailment

Vu, Hoa Trong; Greco, Claudio; Erofeeva, Aliia; Jafaritazehjan, Somayeh; Linders, Guido M.; Tanti, Marc; Testoni, Alberto; Bernardi, Raffaella; Gatt, Albert

doi:10.48550/arxiv.1806.05645

Cited by 5 publications

(9 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This baseline verifies the existing data bias in the SNLI dataset, as mentioned by Gururangan et al [24] and Vu et al [71], by using hypotheses only without the image premise information.…”

Section: Hypothesis Onlysupporting

confidence: 77%

See 1 more Smart Citation

Visual Entailment: A Novel Task for Fine-Grained Image Understanding

Xie,

Lai,

Doran

et al. 2019

Preprint

107

View full text Add to dashboard Cite

Existing visual reasoning datasets such as Visual Question Answering (VQA), often suffer from biases conditioned on the question, image or answer distributions. The recently proposed CLEVR dataset addresses these limitations and requires fine-grained reasoning but the dataset is synthetic and consists of similar objects and sentence structures across the dataset.In this paper, we introduce a new inference task, Visual Entailment (VE) -consisting of image-sentence pairs whereby a premise is defined by an image, rather than a natural language sentence as in traditional Textual Entailment tasks. The goal of a trained VE model is to predict whether the image semantically entails the text. To realize this task, we build a dataset SNLI-VE based on the Stanford Natural Language Inference corpus and Flickr30k dataset. We evaluate various existing VQA baselines and build a model called Explainable Visual Entailment (EVE) system to address the VE task. EVE achieves up to 71% accuracy and outperforms several other state-of-the-art VQA based models. Finally, we demonstrate the explainability of EVE through cross-modal attention visualizations. The SNLI-VE dataset is publicly available at https://github.com/ necla-ml/SNLI-VE.

show abstract

“…This baseline verifies the existing data bias in the SNLI dataset, as mentioned by Gururangan et al [24] and Vu et al [71], by using hypotheses only without the image premise information.…”

Section: Hypothesis Onlysupporting

confidence: 77%

“…Our paper extends the TE task in the visual domain -allowing future work on our SNLI-VE task to build new models on recent progress in SNLI and VQA. Our work is different from the recent work [71] that combines both images and captions as premises.…”

Section: Related Workmentioning

confidence: 83%

Visual Entailment: A Novel Task for Fine-Grained Image Understanding

Xie,

Lai,

Doran

et al. 2019

Preprint

107

View full text Add to dashboard Cite

show abstract

“…Hypothesis Only: This model uses hypotheses only without image premises. Based on no premises, the model was expected to make random guesses but the resulting accuracy is up to 67%, as reproduced by others [Gururangan et al, 2018, Vu et al, 2018. This indicates the performance of our model must exceed the 67% lower bound to make sense.…”

Section: The Eve Architecturementioning

confidence: 60%

“…There is a chance that neutral and contradiction relations may change because the images may include other entities that unexpectedly rewrite neutral and contradiction conclusions. Recently work [Vu et al, 2018] combining both images and captions as premises validates that the effects of conclusion changes happen to be tolerable.…”

Section: Introductionmentioning

confidence: 86%

Visual Entailment Task for Visually-Grounded Language Learning

Xie¹,

Lai²,

Kadav³

2018

Preprint

View full text Add to dashboard Cite

We introduce a new inference task -Visual Entailment (VE) -which differs from traditional Textual Entailment (TE) tasks whereby a premise is defined by an image, rather than a natural language sentence as in TE tasks. A novel dataset SNLI-VE (publicly available at https://github.com/necla-ml/SNLI-VE) is proposed for VE tasks based on the Stanford Natural Language Inference corpus and Flickr30k. We introduce a differentiable architecture called the Explainable Visual Entailment model (EVE) to tackle the VE problem. EVE and several other state-of-the-art visual question answering (VQA) based models are evaluated on the SNLI-VE dataset, facilitating grounded language understanding and providing insights on how modern VQA based models perform.

show abstract

“…We conducted hypothesis only reliance test by using hypothesis only information to train a model as baseline. This is a commonly adopted approach [71,72] in SNLI/RTE to verify the presence of data bias. The assumption is that without any premise information, this baseline is supposed to make a random guess out of the five classes.…”

Section: Hypothesis Only Testmentioning

confidence: 99%

Logically at Factify 2022: Multimodal Fact Verification

Gao¹,

Hoffmann²,

Oikonomou³

et al. 2021

Preprint

View full text Add to dashboard Cite

This paper describes our participant system for the multi-modal fact verification (Factify) challenge at AAAI 2022. Despite the recent advance in text based verification techniques and large pre-trained multimodal models cross vision and language, very limited work has been done in applying multimodal techniques to automate fact checking process, particularly considering the increasing prevalence of claims and fake news about images and videos on social media. In our work, the challenge is treated as multimodal entailment task and framed as multi-class classification. Two baseline approaches are proposed and explored including an ensemble model (combining two uni-modal models) and a multimodal attention network (modeling the interaction between image and text pair from claim and evidence document). We conduct several experiments investigating and benchmarking different SoTA pre-trained transformers and vision models in this work. Our best model is ranked first in leaderboard which obtains a weighted average F-measure of 0.77 on both validation and test set. Exploratory analysis of dataset is also carried out on the Factify data set and uncovers salient patterns and issues (e.g., word overlapping, visual entailment correlation, source bias) that motivates our hypothesis. Finally, we highlight challenges of the task and multimodal dataset for future research.

show abstract

Grounded Textual Entailment

Cited by 5 publications

References 0 publications

Visual Entailment: A Novel Task for Fine-Grained Image Understanding

Visual Entailment: A Novel Task for Fine-Grained Image Understanding

Visual Entailment Task for Visually-Grounded Language Learning

Logically at Factify 2022: Multimodal Fact Verification

Contact Info

Product

Resources

About