What Robot do I Need? Fast Co-Adaptation of Morphology and Control using Graph Neural Networks

Abstract

The co-adaptation of robot morphology and behaviour becomes increasingly important with the advent of fast 3D-manufacturing methods and efficient deep reinforcement learning algorithms. A major challenge for the application of coadaptation methods to the real world is the simulation-to-reality-gap due to model and simulation inaccuracies. However, prior work focuses primarily on the study of evolutionary adaptation of morphologies exploiting analytical models and (differentiable) simulators with large population sizes, neglecting the existence of the simulation-to-reality-gap and the cost of manufacturing cycles in the real world. This paper presents a new approach combining classic high-frequency deep neural networks with computational expensive Graph Neural Networks for the data-efficient co-adaptation of agents with varying numbers of degrees-of-freedom. Evaluations in simulation show that the new method can co-adapt agents within such a limited number of production cycles by efficiently combining design optimization with offline reinforcement learning, that it allows for the direct application to real-world co-adaptation tasks in future work.Preprint. Under review.

Co-imitation: Learning Design and Behaviour by Imitation

“…Similarly, Ha (2019) proposes to use REINFORCE to optimize policy parameters and design parameters of a population of agents in a joint manner. The co-adaptation method presented by Luck, Amor, and Calandra (2020) improves data-efficiency compared to return-based algorithms by utilizing the critic learned by Soft Actor Critic (SAC) (Haarnoja et al 2018) to query for the expected episodic return of unseen designs during design optimization. While the method we present is closest to the former approach, all discussed co-adaptation methods require access to a reward function, and are thus not capable of co-adapting the behaviour and design of an agent without requiring an engineer to formulate a reward function.…”

“…Figure 5 shows the results in the two HalfCheetah morphology transfer scenarios. To address RQ3, we compare CoIL to two other co-imitation approaches: using the cheetah without morphology adaptation, as well as to using the Q-function method adapted from Luck, Amor, and Calandra (2020). Since this method is designed for the standard reinforcement learning setting, we adapt it to the imitation learning scenario by using SAIL to imitate the expert trajectories, and iteratively optimizing the morphology using the Q-function.…”

“…While the processes operate on different time scales, they are closely interconnected and crucial elements leading to the development of well-performing and highly adapted organisms on earth. While research in robot learning has largely been focused on the aspects of behavioural learning processes, a growing number of works have sought to combine behaviour learning and morphology adaptation for robotics applications via co-adaptation (Luck, Amor, and Calandra 2020;Liao et al 2019;Schaff et al 2019;Ha 2019;Le Goff et al 2022). Earlier works focused primarily on the use of evolutionary optimization techniques (Sims 1994;Pollack et al 2000;Alattas, Patel, and Sobh 2019), but Figure 1: The proposed co-imitation algorithm (centre) is able to faithfully match the gait of human motion capture demonstrations (left) by optimizing both the morphology and behaviour of a simulated humanoid.…”

“…with the advent of deep learning, new opportunities arose for the efficient combination of deep reinforcement learning and evolutionary adaptation (Schaff et al 2019;Luck, Amor, and Calandra 2020;Hallawa et al 2021;Luck, Calandra, and Mistry 2021). In contrast to fixed behaviour primitives or simple controllers with a handful of parameters (Lan et al 2021;Liao et al 2019), deep neural networks allow a much greater range of behaviours given a morphology (Luck, Amor, and Calandra 2020). Existing works in co-adaptation, however, focus on a setting where a reward function is assumed to be known, even though engineering a reward function is a notoriously difficult and error-prone task (Singh et al 2019).…”