Mapping Instructions to Actions in 3D Environments with Visual Goal Prediction

Misra, Dipendra; Bennett, Andrew; Blukis, Valts; Niklasson, Eyvind; Shatkhin, Max; Artzi, Yoav

doi:10.18653/v1/d18-1287

Cited by 125 publications

(133 citation statements)

References 30 publications

Supporting

Mentioning

132

Contrasting

Order By: Relevance

“…when being integrated into a new environment). Our work is most closely related to recent advances in instruction following and visual attention [2], [3], but we do not provide explicit supervision for object detections or classifications. Finally, we will make the assumption that goals are specified by a simple list of object IDs, so as to avoid the ambiguity introduced by natural language commands.…”

Section: … …mentioning

confidence: 99%

EARLY FUSION for Goal Directed Robotic Vision

Walsman

Bisk

Gabriel

et al. 2019

2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

Self Cite

View full text Add to dashboard Cite

Building perceptual systems for robotics which perform well under tight computational budgets requires novel architectures which rethink the traditional computer vision pipeline. Modern vision architectures require the agent to build a summary representation of the entire scene, even if most of the input is irrelevant to the agent's current goal. In this work, we flip this paradigm, by introducing EARLYFUSION vision models that condition on a goal to build custom representations for downstream tasks. We show that these goal specific representations can be learned more quickly, are substantially more parameter efficient, and more robust than existing attention mechanisms in our domain. We demonstrate the effectiveness of these methods on a simulated item retrieval problem that is trained in a fully end-to-end manner via imitation learning.

show abstract

Section: … …mentioning

confidence: 99%

EARLY FUSION for Goal Directed Robotic Vision

Walsman

Bisk

Gabriel

et al. 2019

2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

Self Cite

View full text Add to dashboard Cite

show abstract

“…32] and grounded question answering [3,13,34]. Recently, the problem has been studied in interactive simulated environments where the visual input changes as the agent acts, such as interactive question answering [9, 12, ] and instruction following [25,26]. In contrast, we focus on an interactive environment with real-world observations.…”

Section: Related Work and Datasetsmentioning

confidence: 99%

“…We perform linguistically-driven analysis to two additional navigation datasets: SAIL [21,7] and LANI [25], both using simulated environments. Both datasets include paragraphs segmented into single instructions.…”

Section: B Additional Data Analysismentioning

confidence: 99%

See 1 more Smart Citation

TOUCHDOWN: Natural Language Navigation and Spatial Reasoning in Visual Street Environments

Chen

Suhr

Misra

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Self Cite

293

342

View full text Add to dashboard Cite

We study the problem of jointly reasoning about language and vision through a navigation and spatial reasoning task. We introduce the TOUCHDOWN task and dataset, where an agent must first follow navigation instructions in a real-life visual urban environment, and then identify a location described in natural language to find a hidden object at the goal position. The data contains 9,326 examples of English instructions and spatial descriptions paired with demonstrations. Empirical analysis shows the data presents an open challenge to existing methods, and qualitative linguistic analysis shows that the data displays richer use of spatial reasoning compared to related resources. The environment and data are available at https://touchdown.ai. the dumpster has a blue tarp draped over the end closest to you. touchdown is on the top of the blue tarp on the dumpster.LINGUNET The model correctly predicts the location of Touchdown, putting most of the predicted distribution (green) on the top-left of the dumpster at the center. 3TEXT2CONV The model incorrectly predicts the location of Touchdown to the top of the car on the far right. While some of the probability mass is correctly placed on the dumpster, the pixel with the highest probability is on the car. 3CONCATCONV The model correctly predicts the location of Touchdown. The distribution is heavily concentrated at a couple of nearby pixels. 3CONCAT The prediction is similar to CONCATCONV.3 Figure 9. Three of the models are doing fairly well. Only TEXT2CONV fails to predict the location of Touchdown.turn to your right and you will see a green trash barrel between the two blue benches on the right. click to the base of the green trash barrel to find touchdown.LINGUNET The model accurately predicts the green trash barrel on the right as Touchdown's location. 41TEXT2CONV The model predicts successfully as well. The distribution is focused on a smaller area compared to LIN-GUNET closer to the top of the object. This possibly shows a learned bias towards placing Touchdown on the top of objects that TEXT2CONV is more suceptible to. 41CONCATCONV The model prediction is correct. The distribution is focused on fewer pixels compared to LINGUNET. 41CONCAT The model prediction is correct. Similar to CONCATCONV, it focuses on a few pixels. 41 Figure 10. All the models predict the location of Touchdown correctly. Trash can is a relatively common object that workers use to place Touchdown in the dataset . on your right is a parking garage, there is a red sign with bikes parked out in front of the garage, the bear is on the red sign.LINGUNET The model predicted the location of Touchdown correctly to the red stop sign on the right side. 59TEXT2CONV The model predicts the location of Touchdown correctly. 59CONCATCONV The model predicts the location of Touchdown correctly. 59CONCAT The model predicts the location of Touchdown correctly. Figure 11. All the models predict the location of Touchdown correctly. Reference to a red sign are relatively common in the data (Figure 8) potentially sim...

show abstract

“…By specifying the agent's task with a high-level end-goal, our setup does not assume the requester knows how to accomplish the task before requesting it. This aspect, along with the agent-advisor interaction, distinguishes our setup from instruction-following setups [2,45,44,6,12,13], in which the requester provides the agent with detailed sequential steps to execute a task only at the beginning. Constraint formulation.…”

Section: Vision-based Navigation With Languagebased Assistancementioning

confidence: 99%

Vision-Based Navigation With Language-Based Assistance via Imitation Learning With Indirect Intervention

Nguyen

Dey

Brockett

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

We present Vision-based Navigation with Languagebased Assistance (VNLA), a grounded vision-language task where an agent with visual perception is guided via language to find objects in photorealistic indoor environments. The task emulates a real-world scenario in that (a) the requester may not know how to navigate to the target objects and thus makes requests by only specifying high-level endgoals, and (b) the agent is capable of sensing when it is lost and querying an advisor, who is more qualified at the task, to obtain language subgoals to make progress. To model language-based assistance, we develop a general framework termed Imitation Learning with Indirect Intervention (I3L), and propose a solution that is effective on the VNLA task. Empirical results show that this approach significantly improves the success rate of the learning agent over other baselines in both seen and unseen environments.Our code and data are publicly available at https: /

show abstract

Mapping Instructions to Actions in 3D Environments with Visual Goal Prediction

Cited by 125 publications

References 30 publications

EARLY FUSION for Goal Directed Robotic Vision

EARLY FUSION for Goal Directed Robotic Vision

TOUCHDOWN: Natural Language Navigation and Spatial Reasoning in Visual Street Environments

Vision-Based Navigation With Language-Based Assistance via Imitation Learning With Indirect Intervention

Contact Info

Product

Resources

About