Where Are You? Localization from Embodied Dialog

Hahn, Meera; Krantz, Jacob; Batra, Dhruv; Parikh, Devi; Rehg, James M.; Anderson, Peter

doi:10.18653/v1/2020.emnlp-main.59

Cited by 22 publications

(16 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…However, to the best of our knowledge, these prerequisites have not yet been combined with language and the self in mind. Overall, there is a lack of research methods that regard the self in the area of RL, explicitly making use of language in embodied dialogs (Hahn et al, 2020). This section reviews methods that partly satisfy the requirements but still miss at least one of the desired components.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

The Embodied Crossmodal Self Forms Language and Interaction: A Computational Cognitive Review

Röder¹,

Özdemir²,

Nguyen³

et al. 2021

Front. Psychol.

View full text Add to dashboard Cite

Human language is inherently embodied and grounded in sensorimotor representations of the self and the world around it. This suggests that the body schema and ideomotor action-effect associations play an important role in language understanding, language generation, and verbal/physical interaction with others. There are computational models that focus purely on non-verbal interaction between humans and robots, and there are computational models for dialog systems that focus only on verbal interaction. However, there is a lack of research that integrates these approaches. We hypothesize that the development of computational models of the self is very appropriate for considering joint verbal and physical interaction. Therefore, they provide the substantial potential to foster the psychological and cognitive understanding of language grounding, and they have significant potential to improve human-robot interaction methods and applications. This review is a first step toward developing models of the self that integrate verbal and non-verbal communication. To this end, we first analyze the relevant findings and mechanisms for language grounding in the psychological and cognitive literature on ideomotor theory. Second, we identify the existing computational methods that implement physical decision-making and verbal interaction. As a result, we outline how the current computational methods can be used to create advanced computational interaction models that integrate language grounding with body schemas and self-representations.

show abstract

Section: Methodsmentioning

confidence: 99%

“…However, the authors do not consider language either. We address this gap by examining the challenges of embodied dialogs (Hahn et al, 2020) in the context of the self, combining the presence of language with other input modalities to learn appropriate hierarchical representations.…”

Section: Scientific Rationale and Contribution Of This Reviewmentioning

confidence: 99%

The Embodied Crossmodal Self Forms Language and Interaction: A Computational Cognitive Review

Röder¹,

Özdemir²,

Nguyen³

et al. 2021

Front. Psychol.

View full text Add to dashboard Cite

show abstract

“…Rather, the oracle describes the target scene and let the navigator to find it, which commonly occurs when someone get lost in a new building. Hahn et al [200] propose a LED task (localizing the Observer from dialog history) to realize such a scenario. Based on this scenario, they present a dataset named WHERE ARE YOU [200] that consists of 6k dialogs of two humans.…”

Section: Vdn: Find a Tablementioning

confidence: 99%

“…Hahn et al [200] propose a LED task (localizing the Observer from dialog history) to realize such a scenario. Based on this scenario, they present a dataset named WHERE ARE YOU [200] that consists of 6k dialogs of two humans. Due to the wide application of multi-agent communication systems [201], [202], [203], [204] in real-world, researchers become interested in implementing dialog navigating in physical environments.…”

Section: Vdn: Find a Tablementioning

confidence: 99%

Deep Learning for Embodied Vision Navigation: A Survey

Zhu,

Lee

et al. 2021

Preprint

View full text Add to dashboard Cite

The problem of embodied vision navigation attracts rising attention from the community due to its wide application in autonomous driving, vacuum cleaner, and rescue robot. To accomplish embodied vision navigation requires various intelligent skills, such as exploration, mapping, and planning, visual recognition and reasoning, etc. Moreover, building a robot that observes like a human, thinks like a human, and acts like a human helps the community to know how intelligence really is. 3D simulation technology provides a large scale of data to simulate the real-world environment, which enables researchers to train and test the embodied visual navigation model within it. The robustness of deep learning methods empowered the embodied navigation agents to accomplish diverse tasks. However, embodied navigation is still in its infancy due to a lot of challenges, like learning a robust policy from partially observed visual input, learning to exploration in navigation, accomplishing natural language navigation tasks, adapting to the real-world environment, etc. Recently, numerous works have been proposed to tackle different challenges in this area. To give a promising direction for future research, in this paper, we present a comprehensive review of embodied navigation tasks and the recent progress in deep learning-based methods.

show abstract

“…The release of high-quality 3D building and street captures (Chang et al, 2017;Mirowski et al, 2019;Mehta et al, 2020;Xia et al, 2018;Straub et al, 2019) has galvanized interest in developing embodied navigation agents that can operate in complex human environments. Based on these environments, annotations have been collected for a variety of tasks including navigating to a particular class of object (ObjectNav) , navigating from language instructions aka visionand-language navigation (VLN) (Anderson et al, 2018b;Qi et al, 2020;Ku et al, 2020), and vision-and-dialog navigation (Thomason et al, 2020;Hahn et al, 2020). To date, most of these data collection efforts have required the development of custom annotation tools.…”

Section: Introductionmentioning

confidence: 99%

PanGEA: The Panoramic Graph Environment Annotation Toolkit

Ku¹,

Anderson²,

Tuset³

et al. 2021

Proceedings of the Second Workshop on Advances in Language and Vision Research

Self Cite

View full text Add to dashboard Cite

PanGEA, the Panoramic Graph Environment Annotation toolkit, is a lightweight toolkit for collecting speech and text annotations in photo-realistic 3D environments. PanGEA immerses annotators in a web-based simulation and allows them to move around easily as they speak and/or listen. It includes database and cloud storage integration, plus utilities for automatically aligning recorded speech with manual transcriptions and the virtual pose of the annotators. Out of the box, PanGEA supports two tasks -collecting navigation instructions and navigation instruction followingand it could be easily adapted for annotating walking tours, finding and labeling landmarks or objects, and similar tasks. We share best practices learned from using PanGEA in a 20,000 hour annotation effort to collect the Room-Across-Room dataset. We hope that our open-source annotation toolkit and insights will both expedite future data collection efforts and spur innovation on the kinds of grounded language tasks such environments can support.

show abstract

Where Are You? Localization from Embodied Dialog

Cited by 22 publications

References 24 publications

The Embodied Crossmodal Self Forms Language and Interaction: A Computational Cognitive Review

The Embodied Crossmodal Self Forms Language and Interaction: A Computational Cognitive Review

Deep Learning for Embodied Vision Navigation: A Survey

PanGEA: The Panoramic Graph Environment Annotation Toolkit

Contact Info

Product

Resources

About