Reinforcement Learning and Control of a Lower Extremity Exoskeleton for Squat Assistance

Abstract

A significant challenge for the control of a robotic lower extremity rehabilitation exoskeleton is to ensure stability and robustness during programmed tasks or motions, which is crucial for the safety of the mobility-impaired user. Due to various levels of the user’s disability, the human-exoskeleton interaction forces and external perturbations are unpredictable and could vary substantially and cause conventional motion controllers to behave unreliably or the robot to fall down. In this work, we propose a new, reinforcement learning-based, motion controller for a lower extremity rehabilitation exoskeleton, aiming to perform collaborative squatting exercises with efficiency, stability, and strong robustness. Unlike most existing rehabilitation exoskeletons, our exoskeleton has ankle actuation on both sagittal and front planes and is equipped with multiple foot force sensors to estimate center of pressure (CoP), an important indicator of system balance. This proposed motion controller takes advantage of the CoP information by incorporating it in the state input of the control policy network and adding it to the reward during the learning to maintain a well balanced system state during motions. In addition, we use dynamics randomization and adversary force perturbations including large human interaction forces during the training to further improve control robustness. To evaluate the effectiveness of the learning controller, we conduct numerical experiments with different settings to demonstrate its remarkable ability on controlling the exoskeleton to repetitively perform well balanced and robust squatting motions under strong perturbations and realistic human interaction forces.

Myoelectric Control Systems for Upper Limb Wearable Robotic Exoskeletons and Exosuits—A Systematic Review

“…In the field of wearable robotics, reinforcement learning has been used in prosthetic control (e.g., [ 86 ]), lower limb exoskeleton control (e.g., [ 87 ]), and joint torque estimation (e.g., [ 88 ]). Compared to other types of control methods used in EMGs, the reinforcement learning algorithm reflects the interaction between humans and the environment.…”

A Survey on Design and Control of Lower Extremity Exoskeletons for Bipedal Walking

“…RL uses some forms of trial and error training that teaches machines and robots the model of a system by reinforcing its ideas and establishing a result. They have recently been employed in the control of exoskeletons in [158,159] which demonstrated efficient squatting assistance with human interaction. However, future work is needed to further extend the framework to a variety of human walking patterns.…”

Reducing Squat Physical Effort Using Personalized Assistance From an Ankle Exoskeleton

“…These positive effects have also been shown for squat assistance [17]- [20]. When the squat activity was assisted using an active full lower-body exoskeleton for the hip, knee, and ankle joints [21], this system with reinforcement learning-based control reduced the squat effort needed to maintain balance [22]. To make such squat assistance available for everyday use in occupational or rehabilitation setting, researchers have developed single joint assistance exoskeletons that provided either hip [23], [24], or knee [25]- [28] assistance.…”