Towards Navigation by Reasoning over Spatial Configurations

Zhang, Yue; Guo, Qiang; Kordjamshidi, Parisa

doi:10.18653/v1/2021.splurobonlp-1.5

Cited by 5 publications

(4 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this paper, we focus on spatial reasoning over text which can be described as inferring the implicit 1 spatial relations from explicit relations 2 described in the text. Spatial reasoning plays a crucial role in diverse domains, including language grounding (Liu et al, 2022), navigation (Zhang et al, 2021), and human-robot interaction (Venkatesh et al, 2021). By studying this task, we can analyze both the reading comprehension and logical reasoning capabilities of models.…”

Section: Introductionmentioning

confidence: 99%

SPARTQA: A Textual Question Answering Benchmark for Spatial Reasoning

Mirzaee¹,

Faghihi²,

Ning³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

This paper proposes a question-answering (QA) benchmark for spatial reasoning on natural language text which contains more realistic spatial phenomena not covered by prior work and is challenging for state-of-the-art language models (LM). We propose a distant supervision method to improve on this task. Specifically, we design grammar and reasoning rules to automatically generate a spatial description of visual scenes and corresponding QA pairs. Experiments show that further pretraining LMs on these automatically generated data significantly improves LMs' capability on spatial understanding, which in turn helps to better solve two external datasets, bAbI, and boolQ. We hope that this work can foster investigations into more sophisticated models for spatial reasoning over text.

show abstract

Section: Introductionmentioning

confidence: 99%

SPARTQA: A Textual Question Answering Benchmark for Spatial Reasoning

Mirzaee¹,

Faghihi²,

Ning³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

show abstract

“…In Figure 1(b), the instruction "enter the door" does not help distinguish the target viewpoint from other candidate viewpoints since there are multiple doors and walls in the visual environment. As a result, we hypothesize those types of instructions cause the explicit and fine-grained grounding to be less effective for the VLN task, as appears in (Hong et al, 2020b;Zhang et al, 2021) that use sub-instructions and in (Hong et al, 2020a;Hu et al, 2019;Qi et al, 2020a;Zhang and Kordjamshidi, 2022a) that use object-level representations.…”

Section: Introductionmentioning

confidence: 99%

VLN-Trans: Translator for the Vision and Language Navigation Agent

Zhang¹,

Kordjamshidi²

2023

Preprint

View full text Add to dashboard Cite

Language understanding is essential for the navigation agent to follow instructions. We observe two kinds of issues in the instructions that can make the navigation task challenging: 1. The mentioned landmarks are not recognizable by the navigation agent due to the different vision abilities of the instructor and the modeled agent. 2. The mentioned landmarks are applicable to multiple targets, thus not distinctive for selecting the target among the candidate viewpoints. To deal with these issues, we design a translator module for the navigation agent to convert the original instructions into easy-to-follow sub-instruction representations at each step. The translator needs to focus on the recognizable and distinctive landmarks based on the agent's visual abilities and the observed visual environment. To achieve this goal, we create a new synthetic sub-instruction dataset and design specific tasks to train the translator and the navigation agent. We evaluate our approach on Room2Room (R2R), Room4room (R4R), and Room2Room Last (R2R-Last) datasets and achieve state-of-the-art results on multiple benchmarks.

show abstract

“…Understanding spatial language is important in many applications such as navigation (Zhang and Kordjamshidi, 2022;Zhang et al, 2021;Chen et al, 2019), medical domain Kamel Boulos et al, 2019;Massa et al, 2015), and robotics (Venkatesh et al, 2021;Kennedy et al, 2007). However, few benchmarks have directly focused on comprehending the spatial semantics of the text.…”

Section: Introductionmentioning

confidence: 99%

Transfer Learning with Synthetic Corpora for Spatial Role Labeling and Reasoning

Mirzaee¹,

Kordjamshidi²

2022

Preprint

View full text Add to dashboard Cite

Recent research shows synthetic data as a source of supervision helps pretrained language models (PLM) transfer learning to new target tasks/domains. However, this idea is less explored for spatial language. We provide two new data resources on multiple spatial language processing tasks. The first dataset is synthesized for transfer learning on spatial question answering (SQA) and spatial role labeling (SpRL). Compared to previous SQA datasets, we include a larger variety of spatial relation types and spatial expressions. Our data generation process is easily extendable with new spatial expression lexicons. The second one is a real-world SQA dataset with humangenerated questions built on an existing corpus with SPRL annotations. This dataset can be used to evaluate spatial language processing models in realistic situations. We show pretraining with automatically generated data significantly improves the SOTA results on several SQA and SPRL benchmarks, particularly when the training data in the target domain is small.

show abstract

Towards Navigation by Reasoning over Spatial Configurations

Cited by 5 publications

References 17 publications

SPARTQA: A Textual Question Answering Benchmark for Spatial Reasoning

SPARTQA: A Textual Question Answering Benchmark for Spatial Reasoning

VLN-Trans: Translator for the Vision and Language Navigation Agent

Transfer Learning with Synthetic Corpora for Spatial Role Labeling and Reasoning

Contact Info

Product

Resources

About