My Body is a Cage: the Role of Morphology in Graph-Based Incompatible Control

Abstract

Multitask Reinforcement Learning is a promising way to obtain models with better performance, generalisation, data efficiency, and robustness. Most existing work is limited to compatible settings, where the state and action space dimensions are the same across tasks. Graph Neural Networks (GNN) are one way to address incompatible environments, because they can process graphs of arbitrary size. They also allow practitioners to inject biases encoded in the structure of the input graph. Existing work in graph-based continuous control uses the physical morphology of the agent to construct the input graph, i.e., encoding limb features as node labels and using edges to connect the nodes if their corresponded limbs are physically connected. In this work, we present a series of ablations on existing methods that show that morphological information encoded in the graph does not improve their performance. Motivated by the hypothesis that any benefits GNNs extract from the graph structure are outweighed by difficulties they create for message passing, we also propose AMORPHEUS, a transformer-based approach. Further results show that, while AMORPHEUS ignores the morphological information that GNNs encode, it nonetheless substantially outperforms GNN-based methods.

A Generalist Agent

“…Huang et al (2020) used message passing graph networks to build a single locomotor controller for many simulated 2D walker variants. Kurin et al (2020) showed that transformers can outperform graph based approaches for incompatible (i.e. varying embodiment) control, despite not encoding any morphological inductive biases.…”

Can Wikipedia Help Offline Reinforcement Learning?

“…Arguably, a fundamental bottleneck for pretraining in RL is the difficulty in reusing a single network across vastly different tasks, of distinct observation spaces, action spaces, rewards, scenes, and agent morphologies. Preliminary work explored various aspects of this problem through graph neural networks for morphology generalization (Wang et al, 2018b;Pathak et al, 2019;Chen et al, 2018;Kurin et al, 2020), language for universal reward specification (Jiang et al, 2019;Lynch & Sermanet, 2021;Shridhar et al, 2022), and object-centric action spaces (Zeng et al, 2020;Shridhar et al, 2022;Noguchi et al, 2021). Our work is orthogonal to these as we essentially amortize RL algorithm itself, expressed as sequence modeling with Transformer, instead of specific RL domain information, and can be combined with domain-specific pre-training techniques (Yen- Chen et al, 2020;Lynch & Sermanet, 2021) effortlessly.…”

Soft Robots Learn to Crawl: Jointly Optimizing Design and Control with Sim-to-Real Transfer

“…Similar to goal-conditioned policies, our approach learns a single design-conditioned policy π θ : S × A × Ω → [0, 1] to control all the designs in Ω for the specified task. This idea was proposed in the context of co-optimization [50] as well as for the sub-problem of controlling a set of designs with different morphologies [28,32]. This policy can be trained using any RL algorithm on a mixture of data collected with designs in Ω.…”