Just Don’t Fall: An AI Agent’s Learning Journey Towards Posture Stabilisation

Abstract

Learning to maintain postural balance while standing requires a significant, fine coordination effort between the neuromuscular system and the sensory system. It is one of the key contributing factors towards fall prevention, especially in the older population. Using artificial intelligence (AI), we can similarly teach an agent to maintain a standing posture, and thus teach the agent not to fall. In this paper, we investigate the learning progress of an AI agent and how it maintains a stable standing posture through reinforcement learning. We used the Deep Deterministic Policy Gradient method (DDPG) and the OpenSim musculoskeletal simulation environment based on OpenAI Gym. During training, the AI agent learnt three policies. First, it learnt to maintain the Centre-of-Gravity and Zero-Moment-Point in front of the body. Then, it learnt to shift the load of the entire body on one leg while using the other leg for fine tuning the balancing action. Finally, it started to learn the coordination between the two pre-trained policies. This study shows the potentials of using deep reinforcement learning in human movement studies. The learnt AI behaviour also exhibited attempts to achieve an unplanned goal because it correlated with the set goal (e.g., walking in order to prevent falling). The failed attempts to maintain a standing posture is an interesting by-product which can enrich the fall detection and prevention research efforts.

Refined Continuous Control of DDPG Actors via Parametrised Activation

“…The advantage of the proposed method in the bipedal walking problem and the wide variety of activation functions demonstrated (Figure 7) a promising potential for solving several biomechanics problems where different muscles have different characteristics and response functions, as highlighted in [22]. Applications such as fall detection and prevention [23], ocular motility and the associated cognitive load, and motion sickness [24][25][26][27][28][29][30][31], as well as intent prediction of pedestrians and cyclists [32,33].…”

Biomechanic Posture Stabilisation via Iterative Training of Multi-policy Deep Reinforcement Learning Agents

“…Both solutions, however, do not produce realistic posture sequences and thus the need for self induced motion derived by the desire of an AI agent was highlighted in [10,20,21]. In [22], Hossny and Iskander presented an argument for using an AI agent's failed attempts as a dataset for fall detection.…”

“…This, in return, causes a rapidly changing policy every time step. Hossny and Iskander did address this issue in [22] by introducing a modular design of the actor. In their paper, they separated observation encoding from the policies.…”

“…In this study we expand on Hossny and Iskander's work in [22] to solve the generalisation problem. In doing so, we introduce an iterative training procedure.…”

“…We built a similar modular actor where sub-policy neural networks are allocated and initialised randomly. We also utilised the parameterised tanh activation function to accommodate the characteristic variations of different muscles as suggested in [22,33]. Then, iteratively, we trained each sub-policy for a specified number of simulation steps in two stages.…”