Answering Visual What-If Questions: From Actions to Predicted Scene Descriptions

Wagner, M.; Basevi, Hector; Shetty, Rakshith; Li, Wenbin; Malinowski, Mateusz; Fritz, Mario; Leonardis, Aleš

doi:10.1007/978-3-030-11009-3_32

Cited by 18 publications

(18 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(a benchmark dataset for "physical intelligence") (Wagner et al, 2018) has some similarity with ours. It has synthetically rendered table-top scenes, four types of actions (push, rotate, remove and drop) being performed on an object and what-if questions.…”

Section: Related Workmentioning

confidence: 81%

CLEVR_HYP: A Challenge Dataset and Baselines for Visual Question Answering with Hypothetical Actions over Images

Sampat¹,

Kumar²,

Yang³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

Most existing research on visual question answering (VQA) is limited to information explicitly present in an image or a video. In this paper, we take visual understanding to a higher level where systems are challenged to answer questions that involve mentally simulating the hypothetical consequences of performing specific actions in a given scenario. Towards that end, we formulate a vision-language question answering task based on the CLEVR (Johnson et al., 2017a) dataset. Wethen modify the best existing VQA methods and propose baseline solvers for this task. Finally, we motivate the development of better vision-language models by providing insights about the capability of diverse architectures to perform joint reasoning over image-text modality 1 . * corresponding author 1 Dataset setup scripts and code for baselines are made available at https://github.com/shailaja183/clevr_hyp. For additional details about the dataset creation process, refer supplementary material.

show abstract

Section: Related Workmentioning

confidence: 81%

CLEVR_HYP: A Challenge Dataset and Baselines for Visual Question Answering with Hypothetical Actions over Images

Sampat¹,

Kumar²,

Yang³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

show abstract

“…The research design was descriptive qualitative related to developing physics testlet templates. The templates focused on developing knowledge structures [12] through-provoking deeper context-understanding of what-if questions [13]. The testlet accommodated the context expansion of the problems to reveal more comprehensive understanding.…”

Section: Methodsmentioning

confidence: 99%

Deconstructing Physics Problems Using Testlet Templates

Mindyarto¹,

Sugiyanto²

2021

Proceedings of the 6th Asia-Pacific Education and Science Conference, AECon 2020, 19-20 December 2020, Purwokerto, Indonesia

View full text Add to dashboard Cite

Physics problems have been deconstructed into testlet items using templates. These items have been generated based on the functional requirements of a developed automatic item generator. Empirical testing needs to be conducted to verify the conformity with automatic item generation characteristics. This paper aims to address the extent of template compliance with the generator features. Black-box testing procedure was used to verify the function conformities by exploring the execution of the generator and its input and output. 25 Physics templates were examined. The testlet variants generated from the execution of the generator showed that the template characteristics adhered to the input requirements of the generator. It can be concluded that the validation of the testlet templates by the generator showed that the templates support the automatic item generation.

show abstract

“…Since labeled training examples are not readily available in many domains, researchers have explored approaches that simulate labeled data or use prior knowledge to constrain learning. For instance, physics engines have been used to generate labeled data for training deep networks that predict the movement of objects in response to external forces [17,43,60], or for understanding the physics of scenes [5]. A recurrent neural network (RNN) architecture augmented by arithmetic and logical operations has been used to answer questions about scenes [44], but it used textual information instead of the more informative visual data and did not support reasoning with commonsense knowledge.…”

Section: Related Workmentioning

confidence: 99%

Combining Commonsense Reasoning and Knowledge Acquisition to Guide Deep Learning in Robotics

Sridharan¹,

Mota²

2022

Preprint

View full text Add to dashboard Cite

Algorithms based on deep network models are being used for many pattern recognition and decisionmaking tasks in robotics and AI. Training these models requires a large labeled dataset and considerable computational resources, which are not readily available in many domains. Also, it is difficult to explore the internal representations and reasoning mechanisms of these models. As a step towards addressing the underlying knowledge representation, reasoning, and learning challenges, the architecture described in this paper draws inspiration from research in cognitive systems. As a motivating example, we consider an assistive robot trying to reduce clutter in any given scene by reasoning about the occlusion of objects and stability of object configurations in an image of the scene. In this context, our architecture incrementally learns and revises a grounding of the spatial relations between objects and uses this grounding to extract spatial information from input images. Non-monotonic logical reasoning with this information and incomplete commonsense domain knowledge is used to make decisions about stability and occlusion. For images that cannot be processed by such reasoning, regions relevant to the tasks at hand are automatically identified and used to train deep network models to make the desired decisions. Image regions used to train the deep networks are also used to incrementally acquire previously unknown state constraints that are merged with the existing knowledge for subsequent reasoning. Experimental evaluation performed using simulated and real-world images indicates that in comparison with baselines based just on deep networks, our architecture improves reliability of decision making and reduces the effort involved in training data-driven deep network models.

show abstract

Answering Visual What-If Questions: From Actions to Predicted Scene Descriptions

Cited by 18 publications

References 26 publications

CLEVR_HYP: A Challenge Dataset and Baselines for Visual Question Answering with Hypothetical Actions over Images

CLEVR_HYP: A Challenge Dataset and Baselines for Visual Question Answering with Hypothetical Actions over Images

Deconstructing Physics Problems Using Testlet Templates

Combining Commonsense Reasoning and Knowledge Acquisition to Guide Deep Learning in Robotics

Contact Info

Product

Resources

About