Predicting Future Position From Natural Walking and Eye Movements with Machine Learning

Abstract

The prediction of human locomotion behavior is a complex task based on data from the given environment and the user. In this study, we trained multiple machine learning models to investigate if data from contemporary virtual reality hardware enables long-and short-term locomotion predictions. To create our data set, 18 participants walked through a virtual environment with different tasks. The recorded positional, orientation-and eye-tracking data was used to train an LSTM model predicting the future walking target. We distinguished between short-term predictions of 50ms and longterm predictions of 2.5 seconds. Moreover, we evaluated GRUs, sequence-to-sequence prediction, and Bayesian model weights. Our results showed that the best short-term model was the LSTM using positional and orientation data with a mean error of 5.14 mm. The best long-term model was the LSTM using positional, orientation and eye-tracking data with a mean error of 65.73 cm. Gaze data offered the greatest predictive utility for long-term predictions of short distances. Our findings indicate that an LSTM model can be used to predict walking paths in VR. Moreover, our results suggest that eye-tracking data provides an advantage for this task.

Eye Tracking-based LSTM for Locomotion Prediction in VR

“…Usually eye movements to a behavioral target precede any other motor action [18,32]. Therefore, they present a rich signal for the estimation of action intention [3,6,16,56,57]. However, during walking this does not mean that people lock their gaze on a future target, for example a door, at all times.…”

“…Since our prediction model should be based on user behavior without information about the environment, it needed to use a coordinate system in a reference-frame attached to the user and not to the environment. For the present study, we used a head-fixed coordinate system, which has shown the best results previously (details in [6]).…”

“…Thus, eye tracking significantly improved prediction quality, even if only by a small amount (2.78%). Further analysis showed that using a GRU model did not lead to a significantly better prediction (details in [6]).…”

“…Gaze tracking has recently been incorporated into commercially available head mounted displays (HMD). Although their current quality does not match that of research-grade eye trackers [34,43,45] it appears still possible to use gaze information to improve locomotor predictions of VR users [6,16,56,57]. Since gaze behavior depends to some degree on the task (e.g.…”

A Multifunctional Adaptive and Interactive AI system to support people living with stroke, acquired brain or spinal cord injuries: A study protocol

“…In particular, it will combine sensorimotor inputs recorded from PPC with data regarding eye movements to provide end-users with the control of robotic devices. Furthermore, such human-centered AI will interact and have continuous bidirectional exchanges using gaze and other appropriate bahevioural parameters for action selection in the AI system, natural communication between a user and AI system and mutual learning [ 31 ]. These features ensure that the AI naturally and efficiently controls the neuroprosthesis or the assistive device motor output according to the end user’s intention and real needs.…”