Neuro-algorithmic Policies enable Fast Combinatorial Generalization

Vlastelica, Marin; Rolínek, Michal; Martius, Georg

doi:10.48550/arxiv.2102.07456

Cited by 2 publications

(3 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As shown in Tables 2 and 3, most methods focus on changing the loss function or algorithmic approach. Architectural changes informed by inductive biases are less well studied, with notable examples coming from the work of Cobbe et al (2020b), Singh and Zheng (2021), Tang et al (2020), Vlastelica et al (2021), Zambaldi et al (2019). More work can be done on investigating different architectures, either taking inspiration from supervised learning or creating RL-specific architectures.…”

Section: Future Work On Methods For Zero-shot Generalisationmentioning

confidence: 99%

A Survey of Zero-shot Generalisation in Deep Reinforcement Learning

Kirk

Zhang²,

Grefenstette³

et al. 2023

jair

View full text Add to dashboard Cite

The study of zero-shot generalisation (ZSG) in deep Reinforcement Learning (RL) aims to produce RL algorithms whose policies generalise well to novel unseen situations at deployment time, avoiding overfitting to their training environments. Tackling this is vital if we are to deploy reinforcement learning algorithms in real world scenarios, where the environment will be diverse, dynamic and unpredictable. This survey is an overview of this nascent field. We rely on a unifying formalism and terminology for discussing different ZSG problems, building upon previous works. We go on to categorise existing benchmarks for ZSG, as well as current methods for tackling these problems. Finally, we provide a critical discussion of the current state of the field, including recommendations for future work. Among other conclusions, we argue that taking a purely procedural content generation approach to benchmark design is not conducive to progress in ZSG, we suggest fast online adaptation and tackling RL-specific problems as some areas for future work on methods for ZSG, and we recommend building benchmarks in underexplored problem settings such as offline RL ZSG and reward-function variation.

show abstract

Section: Future Work On Methods For Zero-shot Generalisationmentioning

confidence: 99%

A Survey of Zero-shot Generalisation in Deep Reinforcement Learning

Kirk

Zhang²,

Grefenstette³

et al. 2023

jair

View full text Add to dashboard Cite

show abstract

“…As shown in Table 2, most methods focus on changing the loss function or algorithmic approach. Architectural changes informed by inductive biases are less well studied, with notable examples being [154,155,54,124,91]. More work can be done on investigating different architectures, either taking inspiration from supervised learning or creating RL-specific architectures.…”

Section: Future Work On Methods For Generalisationmentioning

confidence: 99%

“…Higgins et al [15,DARLA] uses β-VAEs [123] to encode the inductive bias of disentanglement into the representations of the policy, improving zero-shot performance on various visual variations. Vlastelica et al [124,NAP] incorporates a black-box shortest-path solver to improve generalisation performance in hard navigation problems. Zambaldi et al [125,91] incorporate a relational inductive bias into the model architecture which aids in generalising along ordinal axes of variation, including extrapolation performance.…”

Section: Handling Differences Between Training and Testingmentioning

confidence: 99%

A Survey of Zero-shot Generalisation in Deep Reinforcement Learning

Kirk¹,

Zhang²,

Grefenstette³

et al. 2021

Preprint

View full text Add to dashboard Cite

The study of generalisation in deep Reinforcement Learning (RL) aims to produce RL algorithms whose policies generalise well to novel unseen situations at deployment time, avoiding overfitting to their training environments. Tackling this is vital if we are to deploy reinforcement learning algorithms in real world scenarios, where the environment will be diverse, dynamic and unpredictable. This survey is an overview of this nascent field. We provide a unifying formalism and terminology for discussing different generalisation problems, building upon previous works. We go on to categorise existing benchmarks for generalisation, as well as current methods for tackling the generalisation problem. Finally, we provide a critical discussion of the current state of the field, including recommendations for future work. Among other conclusions, we argue that taking a purely procedural content generation approach to benchmark design is not conducive to progress in generalisation, we suggest fast online adaptation and tackling RL-specific problems as some areas for future work on methods for generalisation, and we recommend building benchmarks in underexplored problem settings such as offline RL generalisation and reward-function variation.

show abstract

Neuro-algorithmic Policies enable Fast Combinatorial Generalization

Cited by 2 publications

References 11 publications

A Survey of Zero-shot Generalisation in Deep Reinforcement Learning

A Survey of Zero-shot Generalisation in Deep Reinforcement Learning

A Survey of Zero-shot Generalisation in Deep Reinforcement Learning

Contact Info

Product

Resources

About