Sim-to-Real Learning of All Common Bipedal Gaits via Periodic Reward Composition

Abstract

We study the problem of realizing the full spectrum of bipedal locomotion on a real robot with sim-to-real reinforcement learning (RL). A key challenge of learning legged locomotion is describing different gaits, via reward functions, in a way that is intuitive for the designer and specific enough to reliably learn the gait across different initial random seeds or hyperparameters. A common approach is to use reference motions (e.g. trajectories of joint positions) to guide learning. However, finding high-quality reference motions can be difficult and the trajectories themselves narrowly constrain the space of learned motion. At the other extreme, reference-free reward functions are often underspecified (e.g. move forward) leading to massive variance in policy behavior, or are the product of significant reward-shaping via trial-and-error, making them exclusive to specific gaits. In this work, we propose a rewardspecification framework based on composing simple probabilistic periodic costs on basic forces and velocities. We instantiate this framework to define a parametric reward function with intuitive settings for all common bipedal gaits -standing, walking, hopping, running, and skipping. Using this function we demonstrate successful sim-to-real transfer of the learned gaits to the bipedal robot Cassie, as well as a generic policy that can transition between all of the two-beat gaits.

Learning Free Gait Transition for Quadruped Robots Via Phase-Guided Controller

“…Integrating task-specific signals into the observation space is another solution for learning multiple gaits. A recent work from Siekmann et al [28] have shown bipedal controller for multiple gaits by using a cycle time with offsets and a vector ratio as gait representation and periodic reward for training. Compared to naive training approaches, the design of the contact-related periodic reward can capture the major characters of gaits.…”

Learning multiple gaits of quadruped robot using hierarchical reinforcement learning

“…Siekmann et el. proposed a simple reward signal to learn several bipedal gaits considering the periodicity of gait control signal [21]. However, previous approaches are quite limited in robustness and scalability because it is based on manual data generation process and limited heuristics.…”

Learning Quadruped Locomotion Policies Using Logical Rules

“…Some researchers have taken advantage of motion priors in order to encode gait-specific knowledge in a reward function. One popular method for encoding such knowledge in a reward function is to maximize the similarity between the robot's motion and a reference trajectory (Peng et al 2020;Smith et al 2021). While this approach has been successfully demonstrated on real robots, it requires significant manual effort to obtain reference trajectories, and constrains the robot's motion to the given trajectory.…”