Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments

Anderson, Peter J.; Wu, Qi; Teney, Damien; Bruce, Jake; Johnson, Mark P.; Sünderhauf, Niko; Reid, Ian R.; Gould, Stephen Jay; Hengel, Anton van den

doi:10.48550/arxiv.1711.07280

Cited by 8 publications

(9 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To show that the scheduled mechanism is able to provide general improvements, we compare our scheduled RL with vanilla RL and a mix-loss [Ranzato et al, 2015] method on this dataset. We use a similar network architecture as in [Anderson et al, 2017]. Instead of training the agent using only a crossentropy loss to imitate demonstration actions, we introduce a distance-based reward.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Scheduled Policy Optimization for Natural Language Communication with Intelligent Agents

Xiong

Guo²,

Yu³

et al. 2018

Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence

View full text Add to dashboard Cite

We investigate the task of learning to interpret natural language instructions by jointly reasoning with visual observations and language inputs. Unlike current methods which start with learning from demonstrations (LfD) and then use reinforcement learning (RL) to fine-tune the model parameters, we propose a novel policy optimization algorithm which can dynamically schedule demonstration learning and RL. The proposed training paradigm provides efficient exploration and better generalization beyond existing methods. Comparing to existing ensemble models, the best single model based on our proposed method tremendously decreases the execution error by over 50% on a block-world environment. To further illustrate the exploration strategy of our RL algorithm, our paper includes systematic studies on the evolution of policy entropy during training.

show abstract

Section: Discussionmentioning

confidence: 99%

“…Figure 6: Distance error evaluated on the unseen development scenes of Room-to-Room environment.More recently, a new dataset[Anderson et al, 2017] with realistic indoor scenes has been released. This dataset (Room-to-Room) includes 21,567 crowd-sourced natural language instructions and 10,800 panoramic RGB-D images.…”

mentioning

confidence: 99%

Scheduled Policy Optimization for Natural Language Communication with Intelligent Agents

Xiong

Guo²,

Yu³

et al. 2018

Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence

View full text Add to dashboard Cite

show abstract

“…The AI community has built numerous platforms to drive algorithmic advances: the Arcade Learning Environment [13], OpenAI Universe [27], Minecraft-based Malmo [28], maze-based Deep-Mind Lab [29], Doom-based ViZDoom [30], AI2-THOR [31], Matterport3D Simulator [32] and House3D [33]. Several of these environments were created to be powerful 3D sandboxes for developing learning algorithms [28,29,30], while HoME additionally aims to provide a unified platform for multimodal learning in a realistic context (Fig.…”

Section: Related Workmentioning

confidence: 99%

HoME: a Household Multimodal Environment

Brodeur,

Perez,

Anand

et al. 2017

Preprint

View full text Add to dashboard Cite

We introduce HoME: a Household Multimodal Environment for artificial agents to learn from vision, audio, semantics, physics, and interaction with objects and other agents, all within a realistic context. HoME integrates over 45,000 diverse 3D house layouts based on the SUNCG dataset, a scale which may facilitate learning, generalization, and transfer. HoME is an open-source, OpenAI Gym-compatible platform extensible to tasks in reinforcement learning, language grounding, soundbased navigation, robotics, multi-agent learning, and more. We hope HoME better enables artificial agents to learn as humans do: in an interactive, multimodal, and richly contextualized setting.

show abstract

“…In addition to being easier to collect, the firstperson perspective captures rich information about the object appearance, as well as the relationships and interactions between the ego-vehicle and objects in the environment. Due to these advantages, egocentric videos have been directly used in applications such as action recognition [5], [6], navigation [7]- [9], and end-to-end autonomous driving [10]. For trajectory prediction, some work has simulated bird's eye views by projecting egocentric video frames onto the ground plane [2], [3], but these projections can be incorrect due to road irregularities or other sources of distortion, which prevent accurate vehicle position prediction.…”

Section: Introductionmentioning

confidence: 99%

Egocentric Vision-based Future Vehicle Localization for Intelligent Driving Assistance Systems

Yao

Choi

et al. 2019

2019 International Conference on Robotics and Automation (ICRA)

117

View full text Add to dashboard Cite

Predicting the future location of vehicles is essential for safety-critical applications such as advanced driver assistance systems (ADAS) and autonomous driving. This paper introduces a novel approach to simultaneously predict both the location and scale of target vehicles in the first-person (egocentric) view of an ego-vehicle. We present a multi-stream recurrent neural network (RNN) encoder-decoder model that separately captures both object location and scale and pixellevel observations for future vehicle localization. We show that incorporating dense optical flow improves prediction results significantly since it captures information about motion as well as appearance change. We also find that explicitly modeling future motion of the ego-vehicle improves the prediction accuracy, which could be especially beneficial in intelligent and automated vehicles that have motion planning capability. To evaluate the performance of our approach, we present a new dataset of first-person videos collected from a variety of scenarios at road intersections, which are particularly challenging moments for prediction because vehicle trajectories are diverse and dynamic. Code and dataset have been made available at: https: //usa.honda-ri.com/hevi arXiv:1809.07408v2 [cs.CV]

show abstract

Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments

Cited by 8 publications

References 0 publications

Scheduled Policy Optimization for Natural Language Communication with Intelligent Agents

Scheduled Policy Optimization for Natural Language Communication with Intelligent Agents

HoME: a Household Multimodal Environment

Egocentric Vision-based Future Vehicle Localization for Intelligent Driving Assistance Systems

Contact Info

Product

Resources

About