Multimodal Text Style Transfer for Outdoor Vision-and-Language Navigation

Zhu, Wanrong; Wang, Xin; Fu, Tsu-Jui; Yan, An; Narayana, Pradyumna; Sone, Kazoo; Basu, Sugato; Wang, William Yang

doi:10.18653/v1/2021.eacl-main.103

Cited by 23 publications

(35 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For facilitating the experiment, Zhu et al [145] divided the original StreetLearn dataset into a small part, i.e., Manh-50, which mainly covers the Manhattan area with 31K training data. In addition, a baseline Multimodal Text Style Transfer learning approach is proposed to generate style-modified instructions for external resources and address the data scarcity issue.…”

Section: Street View Navigationmentioning

confidence: 99%

Vision-Language Navigation: A Survey and Taxonomy

Wu¹,

Cui²,

Li³

2021

Preprint

View full text Add to dashboard Cite

An agent that can understand natural-language instruction and carry out corresponding actions in the visual world is one of the long-term challenges of Artificial Intelligent (AI). Due to multifarious instructions from humans, it requires the agent can combine natural language to vision and action in unstructured, previously unseen environments. If the instruction given by human is a navigation task, this challenge is called Visual-and-Language Navigation (VLN). It is a booming multidisciplinary field of increasing importance and with extraordinary practicality. Instead of focusing on the details of specific methods, this paper provides a comprehensive survey on VLN tasks and makes a classification carefully according the different characteristics of language instructions in these tasks. According to when the instructions are given, the tasks can be divided into single-turn and multi-turn. For single-turn tasks, we further divided them into goal-orientation and route-orientation based on whether the instructions contain a route. For multi-turn tasks, we divided them into imperative task and interactive task based on whether the agent responses to the instructions. This taxonomy enable researchers to better grasp the key point of a specific task and identify directions for future research.

show abstract

Section: Street View Navigationmentioning

confidence: 99%

Vision-Language Navigation: A Survey and Taxonomy

Wu¹,

Cui²,

Li³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Variations of VLN include indoor navigation [3,33,66,40], street-level navigation [9,53], visionand-dialog navigation [59,74,26], VLN in continuous environments [39], and more. Notwithstanding considerable exploration of pretraining strategies [46,27,50,87], data augmentation approaches [20,21,73], agent architectures and loss functions [86,48,49], existing work in this space considers only model-free approaches. Our aim is to unlock model-based approaches to these tasks, using a visual world model to encode prior commonsense knowledge about human environments and thereby relieve the burden on the agent to learn these regularities.…”

Section: Related Workmentioning

confidence: 99%

Pathdreamer: A World Model for Indoor Navigation

Koh¹,

Lee²,

Yang³

et al. 2021

Preprint

View full text Add to dashboard Cite

People navigating in unfamiliar buildings take advantage of myriad visual, spatial and semantic cues to efficiently achieve their navigation goals. Towards equipping computational agents with similar capabilities, we introduce Pathdreamer, a visual world model for agents navigating in novel indoor environments. Given one or more previous visual observations, Pathdreamer generates plausible high-resolution 360 • visual observations (RGB, semantic segmentation and depth) for viewpoints that have not been visited, in buildings not seen during training. In regions of high uncertainty (e.g. predicting around corners, imagining the contents of an unseen room), Pathdreamer can predict diverse scenes, allowing an agent to sample multiple realistic outcomes for a given trajectory. We demonstrate that Pathdreamer encodes useful and accessible visual, spatial and semantic knowledge about human environments by using it in the downstream task of Vision-and-Language Navigation (VLN). Specifically, we show that planning ahead with Pathdreamer brings about half the benefit of looking ahead at actual observations from unobserved parts of the environment. We hope that Pathdreamer will help unlock model-based approaches to challenging embodied navigation tasks such as navigating to specified objects and VLN.

show abstract

“…Through the years, agents with different model architectures and training mechanisms have been proposed for indoor VLN [1,5,6,9,10,11,14,19,23,31,37,38,39,40,41,46] and outdoor VLN [3,24,26,43,44,47]. Backtranslation eases the urgent problem of data scarcity [5].…”

Section: Introductionmentioning

confidence: 99%

“…Imitation learning and reinforcement learning enhance agents' generalization ability [39,40]. With the rise of BERTbased models, researchers also apply Transformer and pretraining to further improve navigation performance [6,10,47]. While applying new techniques to the navigation agents might boost their performance, we still know little about how agents make each turning decision.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Diagnosing Vision-and-Language Navigation: What Really Matters

Zhu¹,

Qi²,

Narayana³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Vision-and-language navigation (VLN) is a multimodal task where an agent follows natural language instructions and navigates in visual environments. Multiple setups have been proposed, and researchers apply new model architectures or training techniques to boost navigation performance. However, recent studies witness a slow-down in the performance improvements in both indoor and outdoor VLN tasks, and the agents' inner mechanisms for making navigation decisions remain unclear. To the best of our knowledge, the way the agents perceive the multimodal input is under-studied and clearly needs investigations. In this work, we conduct a series of diagnostic experiments to unveil agents' focus during navigation. Results show that indoor navigation agents refer to both object tokens and direction tokens in the instruction when making decisions. In contrast, outdoor navigation agents heavily rely on direction tokens and have a poor understanding of the object tokens. Furthermore, instead of merely staring at surrounding objects, indoor navigation agents can set their sights on objects further from the current viewpoint. When it comes to vision-and-language alignments, many models claim that they are able to align object tokens with certain visual targets, but we cast doubt on the reliability of such alignments.

show abstract

Multimodal Text Style Transfer for Outdoor Vision-and-Language Navigation

Cited by 23 publications

References 35 publications

Vision-Language Navigation: A Survey and Taxonomy

Vision-Language Navigation: A Survey and Taxonomy

Pathdreamer: A World Model for Indoor Navigation

Diagnosing Vision-and-Language Navigation: What Really Matters

Contact Info

Product

Resources

About