Phase-dependent trajectory optimization for CPG-based biped walking using path integral reinforcement learning

Abstract

In this study, we introduce a phase-dependent trajectory optimization method for Central Pattern Generator (CPG)-based biped walking controllers. By exploiting the syn chronization property of the CPG controller, many legged loco motion studies have shown that the CPG-based walking controller is robust against external perturbations and works well in real environments. However, due to the nonlinear dynamic property of the coupled oscillator system composed of the CPG controller and the robot, analytically designing the biped trajectory to satisfy the requirements of a target walking pattern is rather difficult.Therefore, using a nonlinear optimization method is reasonable to improve the walking trajectory. To optimize the walking trajectory, a model-free optimal control method is preferable because precise modeling of the ground contact is difficult. On the other hand, model-free trajectory optimization methods have been considered as quite computationally demanding approach.However, because of recent advances in the nonlinear trajectory optimization method, using the model-free optimization method is now a realistic approach for biped trajectory optimization. We use a path integral reinforcement learning method to improve the biped walking trajectory for CPG-based walking controllers.

Combining learned controllers to achieve new goals based on linearly solvable MDPs

“…In the path integral approach, the linearized Bellman is computed along paths starting from given initial states using sampling methods. The path integral approach has been successfully applied to learning of stochastic policies for robots with large degrees of freedom [25], [26], [27], and it is best suited for optimization around stereotyped motion trajectories. However, an additional learning is needed when a new initial state or a new goal state is given.…”

Biped locomotion - Improvement and adaptation

“…CPG's are usually model-free as they do not take into account the system's dynamics. They smoothly produce stable rhythmic patterns, have a reduced control dimensionality, are robust against external perturbations and can be easily integrated with sensorial feedback [7,8,9]. Nevertheless, hand tuning a set of CPGs is a high-complex process due to its parameters' vast state-space (see [4,5,6,7,8]) and requires detailed knowledge about the system dynamics.…”

“…They smoothly produce stable rhythmic patterns, have a reduced control dimensionality, are robust against external perturbations and can be easily integrated with sensorial feedback [7,8,9]. Nevertheless, hand tuning a set of CPGs is a high-complex process due to its parameters' vast state-space (see [4,5,6,7,8]) and requires detailed knowledge about the system dynamics. Although this dimensionality can be minimized using techniques such as dynamical assumptions, biomechanics and reduction of the controlled DOF's, a considerable amount of data is needed to improve the tuning process [3,5,7].…”

“…Because they do not require an exact model of the system, RL algorithms are more suitable for high-dimensional tasks. However, their convergence to a optimum solution is achieved with a large number of trials [2,3,5,7,8,14,16,17].…”

“…Inspired by biological studies which support the neural coordination between rhythmic movements, Central Pattern Generators (CPG's) are a widely used and effective solution to achieve locomotion [6,7,8]. CPG's are usually model-free as they do not take into account the system's dynamics.…”