Unsupervised Visual Attention and Invariance for Reinforcement Learning

Wang, Xudong; Lian, Long; Yu, Stella X.

doi:10.1109/cvpr46437.2021.00661

Cited by 21 publications

(6 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Some works leverage inductive biases, for example, object-oriented architecture (Kansky et al 2017;Yi et al 2022) and disentangled representation (Higgins et al 2017;Peng et al 2022). Some methods want to eliminate misleading information like temporal information (Raileanu et al 2021;Guo et al 2021) or background (Wang et al 2021). Other methods define the metric of invariance for optimizing the representation, like Bisimulation metrics (Zhang et al 2021) and policy similarity (Agarwal et al 2021).…”

Section: Related Workmentioning

confidence: 99%

Conceptual Reinforcement Learning for Language-Conditioned Tasks

Peng

Zhang

et al. 2023

AAAI

View full text Add to dashboard Cite

Despite the broad application of deep reinforcement learning (RL), transferring and adapting the policy to unseen but similar environments is still a significant challenge. Recently, the language-conditioned policy is proposed to facilitate policy transfer through learning the joint representation of observation and text that catches the compact and invariant information across various environments. Existing studies of language-conditioned RL methods often learn the joint representation as a simple latent layer for the given instances (episode-specific observation and text), which inevitably includes noisy or irrelevant information and cause spurious correlations that are dependent on instances, thus hurting generalization performance and training efficiency. To address the above issue, we propose a conceptual reinforcement learning (CRL) framework to learn the concept-like joint representation for language-conditioned policy. The key insight is that concepts are compact and invariant representations in human cognition through extracting similarities from numerous instances in real-world. In CRL, we propose a multi-level attention encoder and two mutual information constraints for learning compact and invariant concepts. Verified in two challenging environments, RTFM and Messenger, CRL significantly improves the training efficiency (up to 70%) and generalization ability (up to 30%) to the new environment dynamics.

show abstract

Section: Related Workmentioning

confidence: 99%

Conceptual Reinforcement Learning for Language-Conditioned Tasks

Peng

Zhang

et al. 2023

AAAI

View full text Add to dashboard Cite

show abstract

“…Zhang et al (2020) use bisimulation metrics to quantify behavioral similarity between states and learn robust task-relevant representations. Wang et al (2021b) extract, without supervision, the visual foreground to provide background invariant inputs to the policy learner.…”

Section: Related Workmentioning

confidence: 99%

Local Feature Swapping for Generalization in Reinforcement Learning

Bertoin¹,

Rachelson²

2022

Preprint

View full text Add to dashboard Cite

Over the past few years, the acceleration of computing resources and research in deep learning has led to significant practical successes in a range of tasks, including in particular in computer vision. Building on these advances, reinforcement learning has also seen a leap forward with the emergence of agents capable of making decisions directly from visual observations. Despite these successes, the over-parametrization of neural architectures leads to memorization of the data used during training and thus to a lack of generalization. Reinforcement learning agents based on visual inputs also suffer from this phenomenon by erroneously correlating rewards with unrelated visual features such as background elements. To alleviate this problem, we introduce a new regularization technique consisting of channel-consistent local permutations (CLOP) of the feature maps. The proposed permutations induce robustness to spatial correlations and help prevent overfitting behaviors in RL. We demonstrate, on the OpenAI Procgen Benchmark, that RL agents trained with the CLOP method exhibit robustness to visual changes and better generalization properties than agents trained using other state-of-the-art regularization techniques. We also demonstrate the effectiveness of CLOP as a general regularization technique in supervised learning.

show abstract

“…DVRL [35], PlaNet [27], and SLAC [44] predict future observations and rewards given current observation and action. Transporter [41] and VAI [77] train an unsupervised keypoint detector to discovery critical objects in image for control. After the agent is deployed, SSL can be used to continuously improve the policy [28].…”

Section: Related Workmentioning

confidence: 99%

Does Self-supervised Learning Really Improve Reinforcement Learning from Pixels?

Li¹,

Shang²,

Das³

et al. 2022

Preprint

View full text Add to dashboard Cite

We investigate whether self-supervised learning (SSL) can improve online reinforcement learning (RL) from pixels. We extend the contrastive reinforcement learning framework (e.g., CURL) that jointly optimizes SSL and RL losses and conduct an extensive amount of experiments with various self-supervised losses. Our observations suggest that the existing SSL framework for RL fails to bring meaningful improvement over the baselines only taking advantage of image augmentation when the same amount of data and augmentation is used. We further perform an evolutionary search to find the optimal combination of multiple self-supervised losses for RL, but find that even such a loss combination fails to meaningfully outperform the methods that only utilize carefully designed image augmentations. Often, the use of self-supervised losses under the existing framework lowered RL performances. We evaluate the approach in multiple different environments including a real-world robot environment, and confirm that no single self-supervised loss or image augmentation method can dominate all environments and that the current framework for joint optimization of SSL and RL is limited. Finally, we empirically investigate the pretraining framework for SSL + RL and the properties of representations learned with different approaches.Preprint. Under review.

show abstract

Unsupervised Visual Attention and Invariance for Reinforcement Learning

Cited by 21 publications

References 18 publications

Conceptual Reinforcement Learning for Language-Conditioned Tasks

Conceptual Reinforcement Learning for Language-Conditioned Tasks

Local Feature Swapping for Generalization in Reinforcement Learning

Does Self-supervised Learning Really Improve Reinforcement Learning from Pixels?

Contact Info

Product

Resources

About