Hierarchical Cross-Modal Agent for Robotics Vision-and-Language Navigation

Irshad, Muhammad Zubair; Ma, Chih-Yao; Kira, Zsolt

doi:10.1109/icra48506.2021.9561806

Cited by 33 publications

(17 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The action space A available to the agent consists of navigation actions between physical states, and a stop action which determines the end of a solution. Navigation actions can be discrete, e.g., turn left, turn right or move forward (Anderson et al, 2018c;Krantz et al, 2020), as well as continuous (Irshad et al, 2021). Finally, the goal of the agent is to predict a solution ψΦ V LN consisting of a sequence of actions a t ∈ A,t ∈ [1, T ] that closest align to the instruction, and thus, to the true solution ψ Φ V LN .…”

Section: Tasks In Embodied Vision-language Planningmentioning

confidence: 99%

“…VLN is the most established EVLP task: a number of datasets (See Section 4.2) exist in both, indoor (Anderson et al, 2018c;Jain et al, 2019;Krantz et al, 2020;Irshad et al, 2021), and outdoor environments (Hermann et al, 2020;Misra et al, 2018). Overall, VLN models have seen considerable progress in improving the ability to get closer to the goal and to the ground truth trajectory (Fried et al, 2018;Li et al, 2020a;Majumdar et al, 2020;Jain et al, 2019).…”

Section: Tasks In Embodied Vision-language Planningmentioning

confidence: 99%

“…Beyond architecture based approaches, navigation in EVLP problems can be tackled by structuring data into smaller subproblems. Certain datasets (Misra et al, 2018;Shridhar et al, 2020;Suhr et al, 2019;Irshad et al, 2021) use sub-goals, which allow for hierarchical approaches to navigation. Agents can predict higher-level actions which correspond to sub-goals (e.g.…”

Section: Modeling Planningmentioning

confidence: 99%

“…Furthermore, this approach is coupled with Student-Forcing, a method which samples an agent's action from a output probability distribution in order to avoid limiting exploration to only the states included in the expert trajectories (Anderson et al, 2018c). Another commonly used approach is DAgger (Ross et al, 2011a), which trains on aggregated trajectories obtained using expert demonstrations (Blukis et al, 2018a;Krantz et al, 2020;Irshad et al, 2021).…”

Section: Supervised Learningmentioning

confidence: 99%

See 3 more Smart Citations

Core Challenges in Embodied Vision-Language Planning

Francis¹,

Kitamura²,

Labelle³

et al. 2021

Preprint

View full text Add to dashboard Cite

Recent advances in the areas of multimodal machine learning and artificial intelligence (AI) have led to the development of challenging tasks at the intersection of Computer Vision, Natural Language Processing, and Embodied AI. Whereas many approaches and previous survey pursuits have characterised one or two of these dimensions, there has not been a holistic analysis at the center of all three. Moreover, even when combinations of these topics are considered, more focus is placed on describing, e.g., current architectural methods, as opposed to also illustrating high-level challenges and opportunities for the field. In this survey paper, we discuss Embodied Vision-Language Planning (EVLP) tasks, a family of prominent embodied navigation and manipulation problems that jointly use computer vision and natural language. We propose a taxonomy to unify these tasks and provide an in-depth analysis and comparison of the new and current algorithmic approaches, metrics, simulated environments, as well as the datasets used for EVLP tasks. Finally, we present the core challenges that we believe new EVLP works should seek to address, and we advocate for task construction that enables model generalizability and furthers real-world deployment.

show abstract

Section: Tasks In Embodied Vision-language Planningmentioning

confidence: 99%

Section: Tasks In Embodied Vision-language Planningmentioning

confidence: 99%

Section: Modeling Planningmentioning

confidence: 99%

Section: Supervised Learningmentioning

confidence: 99%

See 2 more Smart Citations

Core Challenges in Embodied Vision-Language Planning

Francis¹,

Kitamura²,

Labelle³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…In another direction, recent approaches have relaxed the constraint of discrete traversal in VLN into continuous space ( [24,25,26,27,28]). Here, the agent needs to deal with higher task complexity involving time and space.…”

Section: Vision-and-language Navigation (Vln) and Robot Navigationmentioning

confidence: 99%

LanguageRefer: Spatial-Language Model for 3D Visual Grounding

Roh¹,

Desingh²,

Farhadi³

et al. 2021

Preprint

View full text Add to dashboard Cite

To realize robots that can understand human instructions and perform meaningful tasks in the near future, it is important to develop learned models that can understand referential language to identify common objects in real-world 3D scenes. In this paper, we develop a spatial-language model for a 3D visual grounding problem. Specifically, given a reconstructed 3D scene in the form of a point cloud with 3D bounding boxes of potential object candidates, and a language utterance referring to a target object in the scene, our model identifies the target object from a set of potential candidates. Our spatial-language model uses a transformer-based architecture that combines spatial embedding from boundingbox with a finetuned language embedding from DistilBert [1] and reasons among the objects in the 3D scene to find the target object. We show that our model performs competitively on visio-linguistic datasets proposed by ReferIt3D [2]. We provide additional analysis of performance in spatial reasoning tasks decoupled from perception noise, the effect of view-dependent utterances in terms of accuracy, and view-point annotations for potential robotics applications. Project website: https://sites.google.com/view/language-refer.

show abstract