Learning Memory-Based Control for Human-Scale Bipedal Locomotion

Abstract

Controlling a non-statically stable biped is a difficult problem largely due to the complex hybrid dynamics involved. Recent work has demonstrated the effectiveness of reinforcement learning (RL) for simulation-based training of neural network controllers that successfully transfer to real bipeds. The existing work, however, has primarily used simple memoryless network architectures, even though more sophisticated architectures, such as those including memory, often yield superior performance in other RL domains. In this work, we consider recurrent neural networks (RNNs) for sim-to-real biped locomotion, allowing for policies that learn to use internal memory to model important physical properties. We show that while RNNs are able to significantly outperform memoryless policies in simulation, they do not exhibit superior behavior on the real biped due to overfitting to the simulation physics unless trained using dynamics randomization to prevent overfitting; this leads to consistently better sim-to-real transfer. We also show that RNNs could use their learned memory states to perform online system identification by encoding parameters of the dynamics into memory.

Blind Bipedal Stair Traversal via Sim-to-Real Reinforcement Learning

“…We represent the control policy as an LSTM recurrent neural network [17], with two recurrent hidden layers of dimension 128 each. We opt to use a memory-enabled network because of previous work demonstrating a higher degree of proficiency in handling partially observable environments [18] [16] [19]. For ablation experiments, involving non-memory-based control policies, we use a standard feedforward neural network with two layers of dimension 300, with tanh activation functions, such that the number of parameters is approximately equal to that of the LSTM network.…”

“…During training, we make use of a mirror loss term [21] in order to ensure that the control policy does not learn asymmetric gaits. For recurrent policies, we sample batches of episodes from a replay buffer as in [19], while for feedforward policies we sample batches of timesteps. Each episode is limited to be 300 timesteps, which corresponds to about 7.5 seconds of simulation time.…”

“…This indicator function is likely to be 1 during the interval in which it is active, and likely to be 0 during intervals in which it is not active. The distribution of this binary-valued random function is defined via the Von Mises distribution; for a more comprehensive description, see [19]. In addition, rather than use the actual random variable in the reward we instead opt to use its expectation for more stable learning; see Fig.…”

Learning Neuroplastic Matching of Robot Dynamics in Closed-loop CPGs

“…To quantify the matching of the robot's natural dynamics and the control task dynamics we use the neuroelastic activity as a proxy. If the dynamics do not To optimize and tune the control tasks, different methods 111 have been used such as optimization [41]- [43], self-modeling 112 [44], adaptive CPGs [45]- [47], and machine learning 113 techniques [21], [47]- [52]. For this study, we apply 114 Bayesian optimization [53], [54] to minimize the amount is coupled through mechanical coupling.…”

“…While optimization and learning in simulation are efficient and cheap, the transfer of control policies can be difficult due to the sim2real gap [20], [21], [52]. We examine the transferability of our approach by quantifying the sim2real gap comparing simulation and hardware experiments.…”