Learning Spring Mass Locomotion: Guiding Policies With a Reduced-Order Model

Abstract

In this paper, we describe an approach to achieve dynamic legged locomotion on physical robots which combines existing methods for control with reinforcement learning. Specifically, our goal is a control hierarchy in which highestlevel behaviors are planned through reduced-order models, which describe the fundamental physics of legged locomotion, and lower level controllers utilize a learned policy that can bridge the gap between the idealized, simple model and the complex, full order robot. The high-level planner can use a model of the environment and be task specific, while the lowlevel learned controller can execute a wide range of motions so that it applies to many different tasks. In this letter we describe this learned dynamic walking controller and show that a range of walking motions from reduced-order models can be used as the command and primary training signal for learned policies. The resulting policies do not attempt to naively track the motion (as a traditional trajectory tracking controller would) but instead balance immediate motion tracking with long term stability. The resulting controller is demonstrated on a human scale, unconstrained, untethered bipedal robot at speeds up to 1.2 m/s. This letter builds the foundation of a generic, dynamic learned walking controller that can be applied to many different tasks.

OPT-Mimic: Imitation of Optimized Trajectories for Dynamic Quadruped Behaviors

“…Various methods are proposed to solve trajectory optimization efficiently, including collocation method, e.g., [1], [2], [3], and shooting based method, e.g., [4], [5]. Simplified model such as the single rigid body dynamics model or inverted pendulum can also be used to get approximate solution [6], [7].…”

“…In (15), the position, orientation, and joint rewards are the basic motion imitation rewards with desired values x corresponding to that of the reference motion, whereas the action difference and maximum torque rewards are regularizers designed to mitigate specific sim-to-real issues that we observed. The action difference reward penalizes large differences in actions during consecutive RL environment steps to limit vibration and encourage smooth motions [7]. The maximum torque reward penalizes the maximum joint torque observed across all eight joints over the 20ms integration interval between each step, as current spikes were observed to cause faults in the motor controller hardware during experiments, since the power supply units that we used were not dynamic enough to maintain the required voltage during motions involving large impacts with the ground.…”

FastMimic: Model-based Motion Imitation for Agile, Diverse and Generalizable Quadrupedal Locomotion

“…Because of the complex dynamics of legged robots, several reduced-order dynamic models have been developed for legged robot control. For bipedal robots, the inverted pendulum model and its variants have been wildly used [19], [20], [21], [22]. For quadrupedal robots, simplifying the robot into a single rigid body that is driven by the sum of external forces from stance legs is a reliable approach to control [1], [23], [2].…”

Hybrid Bipedal Locomotion Based on Reinforcement Learning and Heuristics

“…On basis of end-to-end frameworks where the action is completely learned, the sum of model-based control signals and the action of learned policies provide external guidance and avoid pointless blind exploration during training. Low-cost model-based controllers such as central pattern generator [ 19 , 22 , 23 ], model-based gait library [ 24 , 25 , 26 ], and heuristic references [ 27 , 28 ] are often adopted in these approaches to achieve agile real-time controlled locomotion. In these frameworks, NN policies learn the residual between optimal decision and reference given by model-based modules.…”