Learning to Walk: Spike Based Reinforcement Learning for Hexapod Robot Central Pattern Generation

Abstract

Learning to walk -i.e., learning locomotion under performance and energy constraints continues to be a challenge in legged robotics. Methods such as stochastic gradient, deep reinforcement learning (RL) have been explored for bipeds, quadrupeds and hexapods. These techniques are computationally intensive and often prohibitive for edge applications. These methods rely on complex sensors and pre-processing of data, which further increases energy and latency. Recent advances in spiking neural networks (SNNs) promise significant reduction in computing owing to the sparse firing of neuros and has been shown to integrate reinforcement learning mechanisms with biologically observed spike time dependent plasticity (STDP). However, training a legged robot to walk by learning the synchronization patterns of central pattern generators (CPG) in an SNN framework has not been shown. This can marry the efficiency of SNNs with synchronized locomotion of CPG based systemsproviding breakthrough end-to-end learning in mobile robotics. In this paper, we propose a reinforcement based stochastic weight update technique for training a spiking CPG. The whole system is implemented on a lightweight raspberry pi platform with integrated sensors, thus opening up exciting new possibilities.

Trends in the Control of Hexapod Robots: A Survey

“…To discuss the adopted models to control hexapods, considering the information provided in Figure 5a, 54.39% of the studies designed traditional control architectures. The main feature of these controllers is the usage of the kinematic or dynamic models of the robots to evaluate the generated locomotion [7]. From Figure 5b, 55.56% of these systems used kinematic models to describe the actuation of the limbs, because it is only required to establish a connection between the angular position of the joints and the coordinates of the feet.…”

“…Traditional control systems consider the hexapod as a rigid body connected to six robotic manipulators and analyze the actuation and control of each leg individually [7]. Moreover, the locomotion is fully described through kinematic and dynamic models, with the robotic legs considered as open-chained mechanisms.…”

“…Therefore, the implementation of this learning method allows the robot to achieve a higher level of autonomy by not requiring any previous information about the environment or human supervision to learn how to walk. For instance, Lele et al [7] implemented RL in an SNN to generate a stable tripod gait. With this method, the hexapod learned to coordinate its legs without pre-programmed gait sequences.…”

“…The most traditional approaches follow predefined routines to generate and adapt locomotion. This type of control relies on the correct definition of the mathematical models of the robot to evaluate the influence of the surroundings in its internal state [7].…”

Adaptive Locomotion Control of a Hexapod Robot via Bio-Inspired Learning

“…Most of these legged robots have exhibited astonish maneuverabilities in a typically structured environment. Among these legged robots, the hexapod robots have been increasingly attracting attention from scientists and a lot of hexapod robotic prototypes have been successfully developed (Stelzer et al, 2012 ; Li et al, 2019 ; Sartoretti et al, 2019 ; Lele et al, 2020 ; Zhao and Revzen, 2020 ). Even though these hexapod robots in shape look very much like the arthropod that the scientists are animating, such as ants or spiders, the robots developed hitherto are still pretty away from real arthropods.…”