What is Right for Me is Not Yet Right for You: A Dataset for Grounding Relative Directions via Multi-Task Learning

Abstract

Understanding spatial relations is essential for intelligent agents to act and communicate in the physical world. Relative directions are spatial relations that describe the relative positions of target objects with regard to the intrinsic orientation of reference objects. Grounding relative directions is more difficult than grounding absolute directions because it not only requires a model to detect objects in the image and to identify spatial relation based on this information, but it also needs to recognize the orientation of objects and integrate this information into the reasoning process. We investigate the challenging problem of grounding relative directions with endto-end neural networks. To this end, we provide GRiD-3D, a novel dataset that features relative directions and complements existing visual question answering (VQA) datasets, such as CLEVR, that involve only absolute directions. We also provide baselines for the dataset with two established end-toend VQA models. Experimental evaluations show that answering questions on relative directions is feasible when questions in the dataset simulate the necessary subtasks for grounding relative directions. We discover that those subtasks are learned in an order that reflects the steps of an intuitive pipeline for processing relative directions.

PATRON: Perspective-Aware Multitask Model for Referring Expression Grounding Using Embodied Multimodal Cues

“…For example, Kazemzadeh et al (2014) developed a dataset of referring expressions to ground objects in photographs of natural scenes. Lee et al (2022) curated another dataset to develop model for comprehending referring expressions through visual question-answering. One of the crucial limitations of these datasets is that the data samples are curated in non-embodied settings, where human presence and nonverbal gestures are not considered in referring expressions.…”

“…Multimodal Representation Learning: Several multimodal representation learning models have been proposed for various tasks, such as human activity recognition Iqbal 2020, 2021;Samyoun* et al 2022;Islam, Yasar, and Iqbal 2022;Feichtenhofer et al 2019), motion prediction (Yasar*, Islam*, and Iqbal 2022;Yasar and Iqbal 2021), visual-question answering (Lu et al 2019;Li et al 2019), and referring expression comprehension (Yu et al 2016;Mao et al 2016). Existing models for REF predominately use similar cross-attention or self-attention methods to fuse multimodal representations (Goyal et al 2020;Lee et al 2022). However, due to lack of perspective diversity and nonverbal gestures, these models do not explicitly learn the perspective taking and the understanding of human nonverbal interaction necessary to comprehend REF.…”

“…Recent works studied REF and E-REF by designing two separate tasks: a relation grounding task (Goyal et al 2020;Dai, Zhang, and Lin 2017;Zhuang et al 2017;) and an object grounding task (Chen et al 2021;Achlioptas et al 2020;Lee et al 2022;Kazemzadeh et al 2014). In a non-embodied setting, the relation grounding task is defined as determining whether a verbal utterance appropriately describes the spatial relationships between objects in a visual scene.…”