Joint Multimodal Embedding and Backtracking Search in Vision-and-Language Navigation

Abstract

Due to the development of computer vision and natural language processing technologies in recent years, there has been a growing interest in multimodal intelligent tasks that require the ability to concurrently understand various forms of input data such as images and text. Vision-and-language navigation (VLN) require the alignment and grounding of multimodal input data to enable real-time perception of the task status on panoramic images and natural language instruction. This study proposes a novel deep neural network model (JMEBS), with joint multimodal embedding and backtracking search for VLN tasks. The proposed JMEBS model uses a transformer-based joint multimodal embedding module. JMEBS uses both multimodal context and temporal context. It also employs backtracking-enabled greedy local search (BGLS), a novel algorithm with a backtracking feature designed to improve the task success rate and optimize the navigation path, based on the local and global scores related to candidate actions. A novel global scoring method is also used for performance improvement by comparing the partial trajectories searched thus far with a plurality of natural language instructions. The performance of the proposed model on various operations was then experimentally demonstrated and compared with other models using the Matterport3D Simulator and room-to-room (R2R) benchmark datasets.

Survey on Multimodal Transformers for Robots

“…There are many studies on language directed navigation [38], [72], [73], [74], [75], [76], [77], [78], [79], [80], [81], [82], [83]. In the language-directed navigation task, there are many benchmark tasks that have been proposed as visuallanguage-navigation (VLN) (R2R [8] , R4R [34], REVERIE [37], NDH [35], HANNNA [36], Robo-VLN [38] ).…”

“…Next, we describe a model that focuses on pre-training in VLN. Various models [79], [80], [81], [82] use pre-training and fine-tuning for VLN tasks. It has been used to solve the problem of limited amount of data in VLN.…”

Vital information matching in vision-and-language navigation

“…To improve the accuracy of visual language navigation tasks, many scholars (Fried et al, 2018 ; Ma et al, 2019 ; Wang et al, 2019 ; Majumdar et al, 2020 ; Zhu F. et al, 2020 ; Hwang and Kim, 2021 ; Lianbo et al, 2021a , b ) have made lots of contribution. The Speaker–Follower model is proposed by Fried.…”

“…Considering many scholars have proposed different multimodal fusion methods (Fried et al, 2018;Landi et al, 2019;Hwang and Kim, 2021), it still can be improved. There are two main reasons as follows: (1) The recent fusion methods cannot accurately work on the relatively important features of visual and textual information, including landmark objects under the scene and landmark directions under the instructions;…”

“…Considering many scholars have proposed different multi-modal fusion methods (Fried et al, 2018 ; Landi et al, 2019 ; Hwang and Kim, 2021 ), it still can be improved. There are two main reasons as follows: (1) The recent fusion methods cannot accurately work on the relatively important features of visual and textual information, including landmark objects under the scene and landmark directions under the instructions; (2) Facing with complex indoor environment, the internal connections among pairs of perceptual inputs cannot reflect a large amount of useful and helpful information, when these connections are ignored.…”