Learning quadrupedal locomotion over challenging terrain

Abstract

Legged locomotion can extend the operational domain of robots to some of the most challenging environments on Earth. However, conventional controllers for legged locomotion are based on elaborate state machines that explicitly trigger the execution of motion primitives and reflexes. These designs have increased in complexity but fallen short of the generality and robustness of animal locomotion. Here, we present a robust controller for blind quadrupedal locomotion in challenging natural environments. Our approach incorporates proprioceptive feedback in locomotion control and demonstrates zero-shot generalization from simulation to natural environments. The controller is trained by reinforcement learning in simulation. The controller is driven by a neural network policy that acts on a stream of proprioceptive signals. The controller retains its robustness under conditions that were never encountered during training: deformable terrains such as mud and snow, dynamic footholds such as rubble, and overground impediments such as thick vegetation and gushing water. The presented work indicates that robust locomotion in natural environments can be achieved by training in simple domains.

Achieving natural behavior in a robot using neurally inspired hierarchical control

“…In one neurally inspired legged robot, a readout map was trained through reward delivery to transform sensory inputs into modulatory changes to a hierarchically lower dynamic oscillator, such that a simulated robot could complete climbs over difficult terrain (1). In other robots, RL is performed in simulation and the policy is then uploaded to a physical body to successfully generate locomotion (13, 24, 32, 43). RL for locomotion has also been successfully performed online with no simulation (18).…”

“…Every module, with the exception of the burst generators, independently and continuously seeks to match its perception with its goal value through corrective outputs in real time. Unlike designs that only implement feedback control at the level of joint control and some form of feed-forward computation above that to generate behavior (11,14, 22, 25, 28, 32, 43, 45, 50), our design uses closed loop negative feedback control at every level of the hierarchy. This feature replicates the purposive nature of animal behavior, which is often mistakenly assumed to be a feedforward process whereby a stimulus input is transformed by the nervous system and results in motor output (51).…”

“…Unlike other robot control architectures that perform model-based control and planning (11, 12, 13, 14, 22, 23, 25, 33, 45, 50), our control architecture generates robust and adaptive goal-directed behavior through a simple feedback process requiring no model of the environment, prediction of future states, or learning. Unlike architectures in which behavior is generated by environmental stimuli or internal system dynamics (1, 2, 3, 4, 18, 24, 28, 29, 32, 36, 35, 37, 43, 46), our architecture generates adaptive behavior by automatically achieving continuously changing internal goals in the control hierarchy. The only computations required are operations like subtraction and integration, which can be successfully implemented by analog computing devices.…”

Learning-Based Methods of Perception and Navigation for Ground Vehicles in Unstructured Environments: A Review

“…In the past, several works have been proposed for the in-situ classification of terrain types, based exclusively on proprioceptive data while actually experiencing the traverse on the (potentially dangerous) terrain [ 88 , 89 ]. Among the works reported in the present survey, a recent and more frequently adopted approach consists of using proprioceptive data during training as well as for data labeling [ 34 , 35 , 66 ].…”

A Review of Physics Simulators for Robotic Applications

“…Though there are few challenges targeted towards learning for robotics, learning methods have been applied in a number of other robotics challenges. A learned locomotion controller for a quadrupedal robot was recently deployed in the DARPA Subterranean challenge [124], for example.…”