Learning symmetric and low-energy locomotion

Abstract

Fig. 1. Locomotion Controller trained for different creatures. (a) Biped walking. (b) Quadruped galloping. (c) Hexapod Walking. (d) Humanoid running.Learning locomotion skills is a challenging problem. To generate realistic and smooth locomotion, existing methods use motion capture, finite state machines or morphology-specific knowledge to guide the motion generation algorithms. Deep reinforcement learning (DRL) is a promising approach for the automatic creation of locomotion control. Indeed, a standard benchmark for DRL is to automatically create a running controller for a biped character from a simple reward function [Duan et al. 2016]. Although several different DRL algorithms can successfully create a running controller, the resulting motions usually look nothing like a real runner. This paper takes a minimalist learning approach to the locomotion problem, without the use of motion examples, finite state machines, or morphology-specific knowledge. We introduce two modifications to the DRL approach that, when used together, produce locomotion behaviors that are symmetric, low-energy, and much closer to that of a real person. First, we introduce a new term to the loss function (not the reward function) that encourages symmetric actions. Second, we introduce a new curriculum learning method that provides modulated physical assistance to help the character with left/right balance and forward movement. The algorithm automatically computes appropriate assistance to the character and gradually relaxes this assistance, so that eventually the character learns to move entirely without help. Because our method does not make use of motion capture data, it can be applied to a variety of character morphologies. We demonstrate locomotion controllers for the lower half of a biped, a full humanoid, a quadruped, and a hexapod. Our results show that learned policies are able to produce symmetric, low-energy gaits. In addition,

Sim-to-Real Transfer for Biped Locomotion

“…We use an identical reward function with w v = 10.0, w a = 0.01, w w = 0.005, w t = 0.2 for all of the presented examples. We also use the mirror symmetry loss proposed in [3] during training of PUP, which we found to improve the quality of the learned locomotion gaits. For controlling the robot to walk in different directions, we rotate the robot's coordinate frame such that the desired walking direction is aligned with the positive x-axis in the robot frame.…”

Self-Imitation Learning of Locomotion Movements through Termination Curriculum

“…The key element in these methods is a surrogate objective function that allows for more than one gradient update per data sample. Studies have shown that PPO outperforms TRPO in most cases, which makes it the dominant algorithm used in continuous control [Peng et al 2018a;Yu et al 2018].…”

“…This technique has been used for learning humanoid climbing movements, where the agent learns 1-, 2-, 3-, and 4-limb movements in order (a limb can be either one of agent's hands or feet) . A recent study proposed a continuous curriculum learning method for providing physical assistance to help the character in locomotion movements [Yu et al 2018].…”

Figure Skating Simulation from Video

“…Swimming has been reproduced using a CPG‐based locomotion controller that can generate muscle contraction signals automatically [SLST14]. Recently, Yu et al [YTL18] used reinforcement learning to learn basic motor skills, they applied curriculum learning and used symmetry as a reward. Hu et al [HLL*19] simulated skiing motion given a small set of control inputs by applying skiing techniques.…”