On the Emergence of Whole-Body Strategies From Humanoid Robot Push-Recovery Learning

Abstract

Balancing and push-recovery are essential capabilities enabling humanoid robots to solve complex locomotion tasks. In this context, classical control systems tend to be based on simplified physical models and hard-coded strategies. Although successful in specific scenarios, this approach requires demanding tuning of parameters and switching logic between specifically-designed controllers for handling more general perturbations. We apply model-free Deep Reinforcement Learning for training a general and robust humanoid push-recovery policy in a simulation environment. Our method targets high-dimensional whole-body humanoid control and is validated on the iCub humanoid. Reward components incorporating expert knowledge on humanoid control enable fast learning of several robust behaviors by the same policy, spanning the entire body. We validate our method with extensive quantitative analyses in simulation, including out-of-sample tasks which demonstrate policy robustness and generalization, both key requirements towards real-world robot deployment.

Reactive Stepping for Humanoid Robots using Reinforcement Learning: Application to Standing Push Recovery on the Exoskeleton Atalante

“…They must be strong enough to sometimes require stepping, but pushing too hard would prohibit learning. As suggested in [11], we apply forces of constant magnitude for a short duration periodically on the pelvis, where the orientation is sampled from a spherical distribution. In this work, the pushes are applied every 3s, with a jitter of 2s to not overfit to a fixed push scheme and learn recovering consecutive pushes.…”

“…However, the motion was slow and unnatural, with limited robustness to external forces. Promising results were achieved in simulation by several authors concurrently regarding standing push recovery [11], [28], [29]. Yet, robust locomotion and standing push recovery for humanoid robots using deep RL falls short from expectation on real devices.…”

“…The standard deviation is taken over episodes per batch. consumption or the norm of the motor velocities [11], [28]. However, components in the reward function have no guarantee to be optimized as they are a minor contribution to the actual loss of learning algorithms.…”

“…Despite those potential limitations, a robust gait and push recovery for the bipedal Cassie robot was recently learned in simulation using deep RL and transferred successfully to the real device [10]. Concurrently, several works on standing push recovery for humanoid robots trained in simulation suggest that the approach is promising [11], [12], although the same level of performance has not been achieved on real humanoid robots yet.…”

Soft Actuation and Compliant Mechanisms in Humanoid Robots

“…A significant advance in recent years has been the motor learning of humanoid robots using deep reinforcement learning [31,32]. In these cases, robot simulator is essential because a very large number of trials and unexpected motion patterns are assumed.…”