HoME: a Household Multimodal Environment

Brodeur, Simon; Perez, Ethan; Anand, Ankesh; Golemo, Florian; Celotti, Luca; Strub, Florian; Rouat, Jean; Larochelle, Hugo; Courville, Aaron

doi:10.48550/arxiv.1711.11017

Cited by 31 publications

(42 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Embodied AI Environments. The recent surge of research in embodied perception is greatly driven by new 3D environments [90,4,13,10,77] and simulation platforms, such as iGibsonV1 [89], AI2-THOR [45], and Habitat [71]. Compared with grid-like or game environments [96,70,7,37], these open accessible, photo-realistic platforms bring perception and planning in a close loop and make the training and testing of embodied agents reproducible [17].…”

Section: Related Workmentioning

confidence: 99%

Collaborative Visual Navigation

Wang,

Zhu

et al. 2021

Preprint

View full text Add to dashboard Cite

As a fundamental problem for Artificial Intelligence, multi-agent system (MAS) is making rapid progress, mainly driven by multi-agent reinforcement learning (MARL) techniques. However, previous MARL methods largely focused on grid-world like or game environments; MAS in visuallyrich environments has remained less explored. To narrow this gap and emphasize the crucial role of perception in MAS, we propose a large-scale 3D dataset, CollaVN, for multi-agent visual navigation (MAVN). In CollaVN, multiple agents are entailed to cooperatively navigate across photo-realistic environments to reach target locations. Diverse MAVN variants are explored to make our problem more general. Moreover, a memory-augmented communication framework is proposed. Each agent is equipped with a private, external memory to persistently store communication information. This allows agents to make better use of their past communication information, enabling more efficient collaboration and robust long-term planning. In our experiments, several baselines and evaluation metrics are designed. We also empirically verify the efficacy of our proposed MARL approach across different MAVN task settings.

show abstract

Section: Related Workmentioning

confidence: 99%

Collaborative Visual Navigation

Wang,

Zhu

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Compared to GPS-based navigation, VLN accepts surrounding environments as visual inputs and correlates them with instruction in human language. Most VLN datasets in the past are based on synthesized 3-D scenes (Kolve et al 2017;Brodeur et al 2017;Wu et al 2018;Yan et al 2018;Song et al 2017).…”

Section: Vision-and-language Navigation Datasetsmentioning

confidence: 99%

Explore the Potential Performance of Vision-and-Language Navigation Model: a Snapshot Ensemble Method

Qin¹,

Misu²,

Wijaya³

2021

Preprint

View full text Add to dashboard Cite

Vision-and-Language Navigation (VLN) is a challenging task in the field of artificial intelligence. Although massive progress has been made in this task over the past few years attributed to breakthroughs in deep vision and language models, it remains tough to build VLN models that can generalize as well as humans. In this paper, we provide a new perspective to improve VLN models. Based on our discovery that snapshots of the same VLN model behave significantly differently even when their success rates are relatively the same, we propose a snapshot-based ensemble solution that leverages predictions among multiple snapshots. Constructed on the snapshots of the existing state-of-the-art (SOTA) model BERT and our past-action-aware modification, our proposed ensemble achieves the new SOTA performance in the R2R dataset challenge in Navigation Error (NE) and Success weighted by Path Length (SPL).

show abstract

“…To help the research in embodied AI learning, various simulated environments have spurred for the community's benefit. Those 3D environments are created from either synthetic scenes [21,44,[47][48][49][50] or real photographs [51][52][53][54]; some of them use game engines to enable physical interactions [21,44,[55][56][57]. In this paper, we choose AI2-THOR, which uses Unity as the physics engine, and build the environment on it.…”

Section: Simulated Environmentsmentioning

confidence: 99%

Communicative Learning with Natural Gestures for Embodied Navigation Agents with Human-in-the-Scene

Wu¹,

Wu²,

Zhu³

et al. 2021

Preprint

View full text Add to dashboard Cite

Human-robot collaboration is an essential research topic in artificial intelligence (AI), enabling researchers to devise cognitive AI systems and affords an intuitive means for users to interact with the robot. Of note, communication plays a central role. To date, prior studies in embodied agent navigation have only demonstrated that human languages facilitate communication by instructions in natural languages. Nevertheless, a plethora of other forms of communication is left unexplored. In fact, human communication originated in gestures and oftentimes is delivered through multimodal cues, e.g., "go there" with a pointing gesture. To bridge the gap and fill in the missing dimension of communication in embodied agent navigation, we propose investigating the effects of using gestures as the communicative interface instead of verbal cues. Specifically, we develop a VR-based 3D simulation environment, named Gesture-based THOR (Ges-THOR), based on AI2-THOR platform. In this virtual environment, a human player is placed in the same virtual scene and shepherds the artificial agent using only gestures. The agent is tasked to solve the navigation problem guided by natural gestures with unknown semantics; we do not use any predefined gestures due to the diversity and versatile nature of human gestures. We argue that learning the semantics of natural gestures is mutually beneficial to learning the navigation task-learn to communicate and communicate to learn. In a series of experiments, we demonstrate that human gesture cues, even without predefined semantics, improve the object-goal navigation for an embodied agent, outperforming various state-of-the-art methods.

show abstract

HoME: a Household Multimodal Environment

Cited by 31 publications

References 25 publications

Collaborative Visual Navigation

Collaborative Visual Navigation

Explore the Potential Performance of Vision-and-Language Navigation Model: a Snapshot Ensemble Method

Communicative Learning with Natural Gestures for Embodied Navigation Agents with Human-in-the-Scene

Contact Info

Product

Resources

About