ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation

Gan, Chuang; Schwartz, J.; Alter, Seth; Schrimpf, Martin; Traer, James; Freitas, Julian De; Kubilius, Jonas; Bhandwaldar, Abhishek; Haber, Nick; Sano, Megumi; Kim, Kuno; Wang, Elias; Mrowca, Damian; Lingelbach, Michael; Curtis, Aidan; Feigelis, Kevin T.; Bear, David M.; Gutfreund, Dan; Cox, David D.; DiCarlo, James J.; McDermott, Josh H.; Tenenbaum, Joshua B.; Yamins, Daniel

doi:10.48550/arxiv.2007.04954

Cited by 55 publications

(80 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recent progress in Embodied Artificial Intelligence, spans both simulation environments Kolve et al (2017); Li et al (2021); Savva et al (2019); Gan et al (2020); Puig et al (2018) and sophisticated tasks Das et al (2018); Anderson et al (2018); Shridhar et al (2020). Our work is most closely related to research in language-guided task completion, Neural SLAM, and exploration.…”

Section: Related Workmentioning

confidence: 99%

Learning to Act with Affordance-Aware Multimodal Neural SLAM

Jia¹,

Lin²,

Zhao³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recent years have witnessed an emerging paradigm shift toward embodied artificial intelligence, in which an agent must learn to solve challenging tasks by interacting with its environment. There are several challenges in solving embodied multimodal tasks, including long-horizon planning, vision-and-language grounding, and efficient exploration. We focus on a critical bottleneck, namely the performance of planning and navigation. To tackle this challenge, we propose a Neural SLAM approach that, for the first time, utilizes several modalities for exploration, predicts an affordance-aware semantic map, and plans over it at the same time. This significantly improves exploration efficiency, leads to robust long-horizon planning, and enables effective vision-and-language grounding. With the proposed Affordance-aware Multimodal Neural SLAM (AMSLAM) approach, we obtain more than 40% improvement over prior published work on the ALFRED benchmark and set a new state-of-the-art generalization performance at a success rate of 23.48% on the test unseen scenes.

show abstract

Section: Related Workmentioning

confidence: 99%

Learning to Act with Affordance-Aware Multimodal Neural SLAM

Jia¹,

Lin²,

Zhao³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Teleoperation for MM is not easy to implement, as it requires enabling the user to control both navigation and manipulation, possibly simultaneously. Previous work on this problem has explored using online click-through interfaces [18,19,20], muscle signals [21,22], joysticks [23,24,25], tablets [26], and virtual reality interfaces [27,28,29,30,31,32,33,34]. These approaches are either scalable or easy to use: web-based tools are widely available but are not well suited to demonstrate dexterous continuous control, while VR and other interfaces are intuitive to use but are not widely available.…”

Section: Related Workmentioning

confidence: 99%

Error-Aware Imitation Learning from Teleoperation Data for Mobile Manipulation

Wong¹,

Tung²,

Kurenkov³

et al. 2021

Preprint

View full text Add to dashboard Cite

In mobile manipulation (MM), robots can both navigate within and interact with their environment and are thus able to complete many more tasks than robots only capable of navigation or manipulation. In this work, we explore how to apply imitation learning (IL) to learn continuous visuo-motor policies for MM tasks. Much prior work has shown that IL can train visuo-motor policies for either manipulation or navigation domains, but few works have applied IL to the MM domain. Doing this is challenging for two reasons: on the data side, current interfaces make collecting high-quality human demonstrations difficult, and on the learning side, policies trained on limited data can suffer from covariate shift when deployed. To address these problems, we first propose MOBILE MANIPULATION ROBOTURK (MOMART), a novel teleoperation framework allowing simultaneous navigation and manipulation of mobile manipulators, and collect a first-of-its-kind large scale dataset in a realistic simulated kitchen setting. We then propose a learned error detection system to address covariate shift by detecting when an agent is in a potential failure state. We train performant IL policies and error detectors from this data, and achieve over 45% task success rate and 85% error detection success rate across multiple multi-stage tasks when trained on expert data. Codebase, datasets, visualization, and more available at https://sites.google.com/view/il-for-mm/home.

show abstract

“…The generation of synthetic data is popular for various computer vision tasks e.g. [4,5]. The goal of 3D rendering is usually to simulate the 3D world closely and accurately using high-quality assets (e.g.…”

Section: Rendering and Procedural Meshesmentioning

confidence: 99%

Beyond Flatland: Pre-training with a Strong 3D Inductive Bias

Gupta¹,

O'Connell²,

Egger³

2021

Preprint

View full text Add to dashboard Cite

Pre-training on large-scale databases consisting of natural images and then finetuning them to fit the application at hand, or transfer-learning, is a popular strategy in computer vision. However, Kataoka et al., 2020 [1] introduced a technique to eliminate the need for natural images in supervised deep learning by proposing a novel synthetic, formula-based method to generate 2D fractals as training corpus. Using one synthetically generated fractal for each class, they achieved transfer learning results comparable to models pre-trained on natural images. In this project, we take inspiration from their work and build on this idea-using 3D procedural object renders. Since the image formation process in the natural world is based on its 3D structure, we expect pre-training with 3D mesh renders to provide an implicit bias leading to better generalization capabilities in a transfer learning setting and that invariances to 3D rotation and illumination are easier to be learned based on 3D data. Similar to the previous work, our training corpus will be fully synthetic and derived from simple procedural strategies; we will go beyond classic data augmentation and also vary illumination and pose which are controllable in our setting and study their effect on transfer learning capabilities in context to prior work. In addition, we will compare the 2D fractal and 3D procedural object networks to human and non-human primate brain data to learn more about the 2D vs. 3D nature of biological vision.35th Pre-registration workshop (NeurIPS 2021), Remote.

show abstract

ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation

Cited by 55 publications

References 37 publications

Learning to Act with Affordance-Aware Multimodal Neural SLAM

Learning to Act with Affordance-Aware Multimodal Neural SLAM

Error-Aware Imitation Learning from Teleoperation Data for Mobile Manipulation

Beyond Flatland: Pre-training with a Strong 3D Inductive Bias

Contact Info

Product

Resources

About