Adaptation to environmental change using reinforcement learning for robotic salamander

CPG-ACTOR: Reinforcement Learning for Central Pattern Generators

“…Researchers applied RL to optimise CPGs in different scenarios [19]- [22]. The common factor among them is the formulation of the actor-critic method; yet, they include the CPG controller in the environment -as depicted in Fig.…”

“…According to the authors [22], the motivations for including CPGs in the environment are their intrinsic recurrent nature and the amount of time necessary to train them, since CPGs have been considered Recurrent Neural Networks (RNNs) (which are computationally expensive and slow to train). In [19], [20] during training and inference, the policy outputs a new set of parameters for the CPGs in response to observations from the environment at every time-step. In this case, the observations processed by the actor network -which usually represent the feedbackare responsible for producing a meaningful set of CPGparameters for the current state.…”

“…CPG-ACTOR is compared against [19] using the same environment. Both the approaches resort to an actor-critic formulation, precisely running the same critic network with two hidden layers of 64 units each.…”

“…Indeed, the main difference is the actor, which is described in detail in Sec. II for the CPG-ACTOR case, while [19] relies on a network with two hidden layers of 64 units each.…”

“…As Sec. IV illustrates, an appropriate comparison between CPG-ACTOR and [19] required the latter to be warm-started to generate desired positions resulting in visible motions of the leg. Differently from the salamander [12], already tuned parameters are not available for the hopping task, hence a meaningful set from [15] was used as reference.…”