The Unsurprising Effectiveness of Pre-Trained Vision Models for Control

Parisi, Simone; Rajeswaran, Aravind; Purushwalkam, Senthil; Gupta, Abhinav

doi:10.48550/arxiv.2203.03580

Cited by 13 publications

(20 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They apply it to VirtualHome and BabyAI tasks, and find that the inclusion of the pretrained language model improves generalisation to novel tasks. Similarly, Parisi et al (2022) demonstrate that vision models pretrained with self-supervised learning, especially crop segmentations and momentum contrast , can be effectively incorporated into control policies.…”

Section: Related Workmentioning

confidence: 92%

A Generalist Agent

Reed¹,

Żołna²,

Parisotto³

et al. 2022

Preprint

View full text Add to dashboard Cite

Inspired by progress in large-scale language modeling, we apply a similar approach towards building a single generalist agent beyond the realm of text outputs. The agent, which we refer to as Gato, works as a multi-modal, multi-task, multi-embodiment generalist policy. The same network with the same weights can play Atari, caption images, chat, stack blocks with a real robot arm and much more, deciding based on its context whether to output text, joint torques, button presses, or other tokens. In this report we describe the model and the data, and document the current capabilities of Gato.

show abstract

Section: Related Workmentioning

confidence: 92%

A Generalist Agent

Reed¹,

Żołna²,

Parisotto³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Unsupervised representation learning for visual control Following the work of Jaderberg et al [64] that demonstrated the effectiveness of auxiliary unsupervised objectives for RL, a variety of unsupervised learning objectives have been studied, including future latent reconstruction [27,65,66,67], bisimulation [68,69], contrastive learning [70,71,72,73,74,30,75,29,76,77], world model learning [8,9,10,54] and reconstruction [78,79]. Recent approaches have also demonstrated that simple data augmentations can sometimes be effective even without such representation learning objectives [80,81,82].…”

Section: Discussionmentioning

confidence: 99%

Masked World Models for Visual Control

Seo¹,

Hafner²,

Líu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Visual model-based reinforcement learning (RL) has the potential to enable sample-efficient robot learning from visual observations. Yet the current approaches typically train a single model end-to-end for learning both visual representations and dynamics, making it difficult to accurately model the interaction between robots and small objects. In this work, we introduce a visual model-based RL framework that decouples visual representation learning and dynamics learning. Specifically, we train an autoencoder with convolutional layers and vision transformers (ViT) to reconstruct pixels given masked convolutional features, and learn a latent dynamics model that operates on the representations from the autoencoder. Moreover, to encode task-relevant information, we introduce an auxiliary reward prediction objective for the autoencoder. We continually update both autoencoder and dynamics model using online samples collected from environment interaction. We demonstrate that our decoupling approach achieves state-of-the-art performance on a variety of visual robotic tasks from Meta-world and RLBench, e.g., we achieve 81.7% success rate on 50 visual robotic manipulation tasks from Meta-world, while the baseline achieves 67.9%. Code is available on the project website: https://sites.google.com/view/mwm-rl.

show abstract

“…Pretrained CLIP features have been used in a number of recent robotics papers to speed up control and navigation tasks. The features can condition the policy network [Khandelwal et al, 2021], or they can be fused throughout the visual encoder to integrate semantic information about the environment [Parisi et al, 2022]. Pretrained language models have also been shown to provide useful initializations from which to train policies to imitate offline trajectories [Reid et al, 2022, Li et al, 2022.…”

Section: Related Workmentioning

confidence: 99%

“…A comparison between pretrained vision-language representations and pretrained ImageNet representations would also be of interest. Because ImageNet representations are optimized for classification rather than encoding objects and relationship, we expect that ImageNet representations would perform more poorly, as shown by Du et al [2021] and Parisi et al [2022].…”

Section: Number Of Objects Held By Lse-ngu Variantsmentioning

confidence: 99%

Semantic Exploration from Language Abstractions and Pretrained Representations

Tam¹,

Rabinowitz²,

Lampinen³

et al. 2022

Preprint

View full text Add to dashboard Cite

Continuous first-person 3D environments pose unique exploration challenges to reinforcement learning (RL) agents, because of their high-dimensional state and action spaces. These challenges can be ameliorated by using semantically meaningful state abstractions to define novelty for exploration. We propose that learned representations shaped by natural language provide exactly this form of abstraction. In particular, we show that vision-language representations, when pretrained on image captioning datasets sampled from the internet, can drive meaningful, task-relevant exploration and improve performance on 3D simulated environments. We also characterize why and how language provides useful abstractions for exploration by comparing the impacts of using representations from a pretrained model, a language oracle, and several ablations. We demonstrate the benefits of our approach in two very different task domains-one that stresses the identification and manipulation of everyday objects, and one that requires navigational exploration in an expansive world-as well as two popular deep RL algorithms: Impala and R2D2. Our results suggest that using language-shaped representations could improve exploration for various algorithms and agents in challenging environments. * Equal contribution Preprint. Under review.

show abstract

The Unsurprising Effectiveness of Pre-Trained Vision Models for Control

Cited by 13 publications

References 0 publications

A Generalist Agent

A Generalist Agent

Masked World Models for Visual Control

Semantic Exploration from Language Abstractions and Pretrained Representations

Contact Info

Product

Resources

About