CLEVR_HYP: A Challenge Dataset and Baselines for Visual Question Answering with Hypothetical Actions over Images

Sampat, Shailaja Keyur; Kumar, Akshay; Yang, Yezhou; Baral, Chitta

doi:10.18653/v1/2021.naacl-main.289

Cited by 9 publications

(2 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Bisk et al (2020) designed the PIQA benchmark to evaluate physical commonsense reasoning in LLMs through question answering. Sampat et al (2021) proposed an extension to the CLEVR dataset, where an agent must reason and answer questions about a scene after a hypothetical action is taken.…”

Section: Related Workmentioning

confidence: 99%

Learning the Effects of Physical Actions in a Multi-modal Environment

Dagan,

Keller,

Lascarides

2023

Findings of the Association for Computational Linguistics: EACL 2023

View full text Add to dashboard Cite

Large Language Models (LLMs) handle physical commonsense information inadequately. As a result of being trained in a disembodied setting, LLMs often fail to predict an action's outcome in a given environment. However, predicting the effects of an action before it is executed is crucial in planning, where coherent sequences of actions are often needed to achieve a goal. Therefore, we introduce the multi-modal task of predicting the outcomes of actions solely from realistic sensory inputs (images and text). Next, we extend an LLM to model latent representations of objects to better predict action outcomes in an environment. We show that multi-modal models can capture physical commonsense when augmented with visual information. Finally, we evaluate our model's performance on novel actions and objects and find that combining modalities help models to generalize and learn physical commonsense reasoning better.

show abstract

Section: Related Workmentioning

confidence: 99%

Learning the Effects of Physical Actions in a Multi-modal Environment

Dagan,

Keller,

Lascarides

2023

Findings of the Association for Computational Linguistics: EACL 2023

View full text Add to dashboard Cite

show abstract

“…Theorem proving (Alvandi and Watt 2019;Zhelezniakov, Zaytsev, and Radyvonenko 2021) and handwritten formula recognition (Dai et al 2019;Sinha et al 2019) are typical cognitive tasks that require perceiving mathematical symbols from pictures and calculate results based on mathematical theorems. CLEVR (Johnson et al 2017;Sampat et al 2021) also provides a benchmark that introduces logical reasoning to visual question answering. Sinha et al (2019) define CLUTRR benchmarks with robustness and generalization evaluations to measure models' perception and reasoning abilities, respectively.…”

Section: Related Work Cognitive Taskmentioning

confidence: 99%

Weakly Supervised Neural Symbolic Learning for Cognitive Tasks

Tian

Chen

et al. 2022

AAAI

View full text Add to dashboard Cite

Despite the recent success of end-to-end deep neural networks, there are growing concerns about their lack of logical reasoning abilities, especially on cognitive tasks with perception and reasoning processes. A solution is the neural symbolic learning (NeSyL) method that can effectively utilize pre-defined logic rules to constrain the neural architecture making it perform better on cognitive tasks. However, it is challenging to apply NeSyL to these cognitive tasks because of the lack of supervision, the non-differentiable manner of the symbolic system, and the difficulty to probabilistically constrain the neural network. In this paper, we propose WS-NeSyL, a weakly supervised neural symbolic learning model for cognitive tasks with logical reasoning. First, WS-NeSyL employs a novel back search algorithm to sample the possible reasoning process through logic rules. This sampled process can supervise the neural network as the pseudo label. Based on this algorithm, we can backpropagate gradients to the neural network of WS-NeSyL in a weakly supervised manner. Second, we introduce a probabilistic logic regularization into WS-NeSyL to help the neural network learn probabilistic logic. To evaluate WS-NeSyL, we have conducted experiments on three cognitive datasets, including temporal reasoning, handwritten formula recognition, and relational reasoning datasets. Experimental results show that WS-NeSyL not only outperforms the end-to-end neural model but also beats the state-of-the-art neural symbolic learning models.

show abstract