Multi-modal Discriminative Model for Vision-and-Language Navigation

Huang, Haoshuo; Jain, Vihan; Mehta, Harsh; Baldridge, Jason; Ie, Eugene

doi:10.48550/arxiv.1905.13358

Cited by 5 publications

(2 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our work constructs a model that can be useful for evaluating and enhancing instruction-generation models. Huang et al (2019) and Zhao et al (2021) train LSTM-based discriminative models with contrastive learning to score instructions. We follow a similar approach but focus on identifying word-level hallucinations, and effectively leverage a large pre-trained Transformer model.…”

Section: Related Workmentioning

confidence: 99%

Hallucination Detection for Grounded Instruction Generation

Zhao,

Nguyen,

Daumé III

2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

We investigate the problem of generating instructions to guide humans to navigate in simulated residential environments. A major issue with current models is hallucination: they generate references to actions or objects that are inconsistent with what a human follower would perform or encounter along the described path. We develop a model that detects these hallucinated references by adopting a model pretrained on a large corpus of image-text pairs, and fine-tuning it with a contrastive loss that separates correct instructions from instructions containing synthesized hallucinations. Our final model outperforms several baselines, including using word probability estimated by the instruction-generation model, and supervised models based on LSTM and Transformer.

show abstract

Section: Related Workmentioning

confidence: 99%

Hallucination Detection for Grounded Instruction Generation

Zhao,

Nguyen,

Daumé III

2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

show abstract

“…Research on the VLN task has made significant progress in the past few years. Attention mechanisms across different modalities are widely used to learn an alignment between vision and language, to boost the performance of this task [1], [7], [11]- [16]. Except for the new modelling architectures, the improvements also come from new learning approaches.…”

Section: Related Workmentioning

confidence: 99%

Find a Way Forward: a Language-Guided Semantic Map Navigator

Wang¹,

Li²,

Wu³

et al. 2022

Preprint

View full text Add to dashboard Cite

This paper attacks the problem of languageguided navigation in a new perspective by using novel semantic navigation maps, which enables robots to carry out natural language instructions and move to a target position based on the map observations. We break down this problem into parts and introduce three different modules to solve the corresponding subproblems. Our approach leverages map information to provide Deterministic Path Candidate Proposals to reduce the solution space. Different from traditional methods that predict robots' movements toward the target step-by-step, we design an attention-based Language Driven Discriminator to evaluate path candidates and determine the best path as the final result. To represent the map observations along a path for a better modality alignment, a novel Path Feature Encoding scheme tailored for semantic navigation maps is proposed. Unlike traditional methods that tend to produce cumulative errors or be stuck in local decisions, our method which plans paths based on global information can greatly alleviate these problems. The proposed approach has noticeable performance gains, especially in long-distance navigation cases. Also, its training efficiency is significantly higher than of other methods.

show abstract

Improving Cross-Modal Alignment in Vision Language Navigation via Syntactic Information

Li¹,

Tan²,

Bansal³

2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

Vision language navigation is the task that requires an agent to navigate through a 3D environment based on natural language instructions. One key challenge in this task is to ground instructions with the current visual information that the agent perceives. Most of the existing work employs soft attention over individual words to locate the instruction required for the next action. However, different words have different functions in a sentence (e.g., modifiers convey attributes, verbs convey actions). Syntax information like dependencies and phrase structures can aid the agent to locate important parts of the instruction. Hence, in this paper, we propose a navigation agent that utilizes syntax information derived from a dependency tree to enhance alignment between the instruction and the current visual scenes. Empirically, our agent outperforms the baseline model that does not use syntax information on the Room-to-Room dataset, especially in the unseen environment. Besides, our agent achieves the new state-of-the-art on Room-Across-Room dataset, which contains instructions in 3 languages (English, Hindi, and Telugu). We also show that our agent is better at aligning instructions with the current visual information via qualitative visualizations. 1

show abstract

Multi-modal Discriminative Model for Vision-and-Language Navigation

Cited by 5 publications

References 13 publications

Hallucination Detection for Grounded Instruction Generation

Hallucination Detection for Grounded Instruction Generation

Find a Way Forward: a Language-Guided Semantic Map Navigator

Improving Cross-Modal Alignment in Vision Language Navigation via Syntactic Information

Contact Info

Product

Resources

About