HOP+: History-Enhanced and Order-Aware Pre-Training for Vision-and-Language Navigation

Qiao, Yanyuan; Qi, Yuankai; Hong, Yicong; Yu, Zheng; Wang, Peng; Wu, Qi

doi:10.1109/tpami.2023.3234243

Cited by 25 publications

(6 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, transformer-based models have shown superior performance thanks to their powerful ability to learn generic multi-modal representations [43], [44], [45]. This scheme is further extended by recurrent agent state [4], [46], [47], episodic memory [5], [48], [49], graph memory [15], [50], [51] and prompt learning [52], [53] that significantly improves sequential action prediction.…”

Section: Vision-language Navigationmentioning

confidence: 99%

Structured Scene Memory for Vision-Language Navigation

Wang

Yang

Liang

et al. 2021

2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

Vision-language navigation is a task that requires an agent to follow instructions to navigate in environments. It becomes increasingly crucial in the field of embodied AI, with potential applications in autonomous navigation, search and rescue, and human-robot interaction. In this paper, we propose to address a more practical yet challenging counterpart setting -vision-language navigation in continuous environments (VLN-CE). To develop a robust VLN-CE agent, we propose a new navigation framework, ETPNav, which focuses on two critical skills: 1) the capability to abstract environments and generate long-range navigation plans, and 2) the ability of obstacle-avoiding control in continuous environments. ETPNav performs online topological mapping of environments by self-organizing predicted waypoints along a traversed path, without prior environmental experience. It privileges the agent to break down the navigation procedure into high-level planning and low-level control. Concurrently, ETPNav utilizes a transformer-based cross-modal planner to generate navigation plans based on topological maps and instructions. The plan is then performed through an obstacle-avoiding controller that leverages a trial-and-error heuristic to prevent navigation from getting stuck in obstacles. Experimental results demonstrate the effectiveness of the proposed method. ETPNav yields more than 10% and 20% improvements over prior state-of-the-art on R2R-CE and RxR-CE datasets, respectively. Our code is available at https://github.com/MarSaKi/ETPNav.

show abstract

Section: Vision-language Navigationmentioning

confidence: 99%

Structured Scene Memory for Vision-Language Navigation

Wang

Yang

Liang

et al. 2021

2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

show abstract

“…The research on VLN is dedicated to addressing the alignment of linguistic instructions with visual cues and actions, some work fine-graining the navigation instructions to achieve sub-goal planning (Hong et al 2020;He et al 2021a;Zhu et al 2020), and some concentrate on utilizing object information to identify landmarks from observations (Gao et al 2021;Qi et al 2020aQi et al , 2021. Temporal information is specifically designed in (Hao et al 2020;Hong et al 2021;Chen et al 2021Chen et al , 2022Qiao et al 2022Qiao et al , 2023Zhao, Qi, and Wu 2023) to capture long-range dependencies across past observations and actions, which are crucial during navigation. some methods incorporate external knowledge during navigation (Li et al 2022;Gao et al 2021).…”

Section: Related Workmentioning

confidence: 99%

WebVLN: Vision-and-Language Navigation on Websites

Chen,

Pitawela,

Zhao

et al. 2024

AAAI

View full text Add to dashboard Cite

Vision-and-Language Navigation (VLN) task aims to enable AI agents to accurately understand and follow natural language instructions to navigate through real-world environments, ultimately reaching specific target locations. We recognise a promising opportunity to extend VLN to a comparable navigation task that holds substantial significance in our daily lives, albeit within the virtual realm: navigating websites on the Internet. This paper proposes a new task named Vision-and-Language Navigation on Websites (WebVLN), where we use question-based instructions to train an agent, emulating how users naturally browse websites. Unlike the existing VLN task that only pays attention to vision and instruction (language), the WebVLN agent further considers underlying web-specific content like HTML, which could not be seen on the rendered web pages yet contain rich visual and textual information. Toward this goal, we contribute a dataset, WebVLN-v1, and introduce a novel approach called Website-aware VLN Network (WebVLN-Net), which is built upon the foundation of state-of-the-art VLN techniques. Experimental results show that WebVLN-Net outperforms current VLN and web-related navigation methods. We believe that the introduction of the newWebVLN task and its dataset will establish a new dimension within the VLN domain and contribute to the broader vision-and-language research community. Code is available at: https://github.com/WebVLN/WebVLN.

show abstract

“…utilizing common-sense knowledge to address the remote embodied visual referring expression in real indoor environments (REVERIE) task [37]. Furthermore, Qiao et al [38] proposed a history-enhanced and order-aware pretraining with the complementing fine-tuning paradigm (HOP+) for visionand-language navigation. The regions of interest (ROI) are extracted from an image modality as image features using the Fast R-CNN algorithm [39].…”

Section: A Vision and Language Pretrained Modelsmentioning

confidence: 99%

Multimodality Self-distillation for Fast Inference of Vision and Language Pretrained Models

Kong,

Wang,

et al. 2024

IEEE Trans. Multimedia

View full text Add to dashboard Cite

The computational cost of the vision and language pretrained models (VL-PTMs) limits their deployment in resource-constrained devices that require low latency. One existing solution is to apply the early exiting (EE) strategy to accelerate the inference. This technique can force model prediction using only a few former transformer layers. However, these former layers behave differently with the final classifier, inevitably resulting in performance decline. To counter such limitation, self-distillation has been commonly introduced to enhance the representation abilities of the EE classifiers. This results in a semantic gap since EE classifiers are directly trained to mimic the outputs of the final classifier without access to the modality-specific behaviors. This study proposes a multimodality self-distillation method for the fast inference of VL-PTMs. To fill the semantic gap between modalities, we split the multimodalities into separate modalities and added them as extra inputs to encourage the effective distillation of each modality. Furthermore, the mean squared error (MSE) is introduced to minimize the distance of feature maps and further enhance the representation ability of the EE classifiers. Experiments show that the proposed method outperforms the previous EE strategies with the same inference time, and performs competitively even if the model exited very early.

show abstract

HOP+: History-Enhanced and Order-Aware Pre-Training for Vision-and-Language Navigation

Cited by 25 publications

References 35 publications

Structured Scene Memory for Vision-Language Navigation

Structured Scene Memory for Vision-Language Navigation

WebVLN: Vision-and-Language Navigation on Websites

Multimodality Self-distillation for Fast Inference of Vision and Language Pretrained Models

Contact Info

Product

Resources

About