Masked Visual Pre-training for Motor Control

Xiao, Tete; Radosavovic, Ilija; Darrell, Trevor; Malik, Jitendra

doi:10.48550/arxiv.2203.06173

Cited by 20 publications

(48 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recent approaches have also demonstrated that simple data augmentations can sometimes be effective even without such representation learning objectives [80,81,82]. The work closest to ours is Xiao et al [83], which demonstrated that frozen representations from MAE pre-trained on real-world videos can be used for training RL agents on visual manipulation tasks. Our work differs in that we demonstrate that training a self-supervised ViT with reconstruction and convolutional feature masking can be more effective for visual control tasks when compared to MAE that masks pixel patches.…”

Section: Discussionmentioning

confidence: 91%

“…Our work differs in that we demonstrate that training a self-supervised ViT with reconstruction and convolutional feature masking can be more effective for visual control tasks when compared to MAE that masks pixel patches. We also note that our work is orthogonal to Xiao et al [83] in that our framework can also initialize the autoencoder with pre-trained parameters.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Masked World Models for Visual Control

Seo¹,

Hafner²,

Líu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Visual model-based reinforcement learning (RL) has the potential to enable sample-efficient robot learning from visual observations. Yet the current approaches typically train a single model end-to-end for learning both visual representations and dynamics, making it difficult to accurately model the interaction between robots and small objects. In this work, we introduce a visual model-based RL framework that decouples visual representation learning and dynamics learning. Specifically, we train an autoencoder with convolutional layers and vision transformers (ViT) to reconstruct pixels given masked convolutional features, and learn a latent dynamics model that operates on the representations from the autoencoder. Moreover, to encode task-relevant information, we introduce an auxiliary reward prediction objective for the autoencoder. We continually update both autoencoder and dynamics model using online samples collected from environment interaction. We demonstrate that our decoupling approach achieves state-of-the-art performance on a variety of visual robotic tasks from Meta-world and RLBench, e.g., we achieve 81.7% success rate on 50 visual robotic manipulation tasks from Meta-world, while the baseline achieves 67.9%. Code is available on the project website: https://sites.google.com/view/mwm-rl.

show abstract

Section: Discussionmentioning

confidence: 91%

Section: Discussionmentioning

confidence: 99%

Masked World Models for Visual Control

Seo¹,

Hafner²,

Líu³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Pre-training representations for RL One common way pre-training has been used in RL is to improve the representations that are used in the policy network. Self-supervised representation learning techniques distill knowledge from external datasets to produce downstream features that are helpful in virtual environments [Du et al, 2021, Xiao et al, 2022. Some recent work shows benefits from pre-training on more general, large-scale datasets.…”

Section: Related Workmentioning

confidence: 99%

Semantic Exploration from Language Abstractions and Pretrained Representations

Tam¹,

Rabinowitz²,

Lampinen³

et al. 2022

Preprint

View full text Add to dashboard Cite

Continuous first-person 3D environments pose unique exploration challenges to reinforcement learning (RL) agents, because of their high-dimensional state and action spaces. These challenges can be ameliorated by using semantically meaningful state abstractions to define novelty for exploration. We propose that learned representations shaped by natural language provide exactly this form of abstraction. In particular, we show that vision-language representations, when pretrained on image captioning datasets sampled from the internet, can drive meaningful, task-relevant exploration and improve performance on 3D simulated environments. We also characterize why and how language provides useful abstractions for exploration by comparing the impacts of using representations from a pretrained model, a language oracle, and several ablations. We demonstrate the benefits of our approach in two very different task domains-one that stresses the identification and manipulation of everyday objects, and one that requires navigational exploration in an expansive world-as well as two popular deep RL algorithms: Impala and R2D2. Our results suggest that using language-shaped representations could improve exploration for various algorithms and agents in challenging environments. * Equal contribution Preprint. Under review.

show abstract

“…In this paper, we study allocation policies for multiple humans and multiple robots. While many existing works [58,59,60,61,62] have leveraged NVIDIA's Isaac Gym's [11] capability of parallel simulation to accelerate reinforcement learning with multiple robots, these approaches consider robots in isolation without interactive human supervision. The work that is closest to ours is by Swamy et al [63], who study the multi-robot, single-human setting of allocating the attention of one human operator during robot fleet learning.…”

Section: Multi-robot Interactive Learningmentioning

confidence: 99%

Fleet-DAgger: Interactive Robot Fleet Learning with Scalable Human Supervision

Hoque¹,

Chen²,

Sharma³

et al. 2022

Preprint

View full text Add to dashboard Cite

Commercial and industrial deployments of robot fleets often fall back on remote human teleoperators during execution when robots are at risk or unable to make task progress. With continual learning, interventions from the remote pool of humans can also be used to improve the robot fleet control policy over time. A central question is how to effectively allocate limited human attention to individual robots. Prior work addresses this in the single-robot, single-human setting. We formalize the Interactive Fleet Learning (IFL) setting, in which multiple robots interactively query and learn from multiple human supervisors. We present a fully implemented open-source IFL benchmark suite of GPU-accelerated Isaac Gym environments for the evaluation of IFL algorithms. We propose Fleet-DAgger, a family of IFL algorithms, and compare a novel Fleet-DAgger algorithm to 4 baselines in simulation. We also perform 1000 trials of a physical block-pushing experiment with 4 ABB YuMi robot arms. Experiments suggest that the allocation of humans to robots significantly affects robot fleet performance, and that our algorithm achieves up to 8.8× higher return on human effort than baselines. See https: //tinyurl.com/fleet-dagger for code, videos, and supplemental material.

show abstract

Masked Visual Pre-training for Motor Control

Cited by 20 publications

References 40 publications

Masked World Models for Visual Control

Masked World Models for Visual Control

Semantic Exploration from Language Abstractions and Pretrained Representations

Fleet-DAgger: Interactive Robot Fleet Learning with Scalable Human Supervision

Contact Info

Product

Resources

About