Neural Network 3D Body Pose Tracking and Prediction for Motion-to-Photon Latency Compensation in Distributed Virtual Reality

Pöhl, Sebastian; Becher, Armin; Grauschopf, Thomas; Axenie, Cristian

doi:10.1007/978-3-030-30508-6_35

Cited by 3 publications

(3 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Previous work also uses classical software-based approaches to estimate future movements [22]. While ANNs are used to predict head and end-effector translation [37], the effects of full-body-related predictions using ANNs and evaluations of system with embodied users, gauging how latency compensation and future predictions are actually perceived by the user, are currently unknown.…”

Section: Discussionmentioning

confidence: 99%

“…The virtual human avatar movement model generated ranged from 25 to 100 ms in the future and subjectively induced a "lighter weight" sensation [22]. ANNs were used to predict head movements and to calculate the translation of a virtual camera in an upcoming frame [39] as well as for inverse kinematics (IK) end-effector movements [37].…”

Section: Movement Predictions In Vrmentioning

confidence: 99%

See 1 more Smart Citation

The Effects of Full-Body Avatar Movement Predictions in Virtual Reality using Neural Networks

Schwind

Fehle

Sasse

et al. 2020

26th ACM Symposium on Virtual Reality Software and Technology

View full text Add to dashboard Cite

Motion tracking technologies and avatars in virtual reality (VR) showing the movements of the own body enable high levels of presence and a strong illusion of body ownership (IBO)-key features of immersive systems and gaming experiences in virtual environments. Previous work suggests using software-based algorithms that can not only compensate system latency but also predict future movements of the user to increase input performance. However, the effects of movement prediction in VR on input performance are largely unknown. In this paper, we investigate neural networkbased predictions of full-body avatar movements in two scenarios: In the first study, we used a standardized 2D Fitts' Law task to examine the information throughput in VR. In the second study, we utilized a full-body VR game to determine the users' performance. We found that both performance and subjective measures in a standardized 2D Fitts' law task could not benefit from the predicted avatar movements. In an immersive gaming scenario, however, the perceived accuracy of the own body location improved. Presence and body assessments remained more stable and were higher than during the Fitts' task. We conclude that machine-learning-based predictions could be used to compensate system-related latency but participants only subjectively benefit under certain conditions.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Movement Predictions In Vrmentioning

confidence: 99%

The Effects of Full-Body Avatar Movement Predictions in Virtual Reality using Neural Networks

Schwind

Fehle

Sasse

et al. 2020

26th ACM Symposium on Virtual Reality Software and Technology

View full text Add to dashboard Cite

show abstract

“…Predicting human motion accurately plays a pivotal role in various fields, such as computer vision, autonomous driving [13,9], intelligent robots [19], human-robot collaboration [32,28], virtual reality [38], and can occur in different environments, such as streets, airports and sports arenas. The task involves two main aspects: 3D poses estimation and 2D trajectories prediction.…”

Section: Introductionmentioning

confidence: 99%

A Dual Perspective of Human Motion Analysis - 3D Pose Estimation and 2D Trajectory Prediction

Zaier,

Wannous,

Drira

et al. 2023

2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)

View full text Add to dashboard Cite

Anticipating human motion based on given sequences is a challenging and crucial task in computer vision and machine learning, enabling machines to understand human behaviors effectively. Precise prediction of human pose and motion trajectory holds great significance for various applications, including autonomous driving, robotics, and virtual reality. This paper presents a novel approach to address the interconnected tasks of estimating human motion, represented as 3D poses or 2D trajectories, and predicting future motions using 2D images and human pose/position sequences jointly. We propose an encoder-decoder architecture that leverages Transformer networks with a selfattention mechanism, utilizing visual context features, combined with an LSTM to model human motion kinematics. Our approach demonstrates consistent and remarkable improvements over existing methods, both quantitatively and qualitatively. Extensive experiments conducted on diverse public datasets, such as GTA-IM and PROX for 3D human pose estimation, and ETH and UCY combined datasets for 2D trajectory prediction, showcase that our method substantially reduces prediction errors compared to the current state-of-the-art methods.

show abstract

Multi-granularity spatial temporal graph convolution network with consecutive attention for human motion prediction

Ma,

Zhang,

Zhou

et al. 2024

Applied Soft Computing

View full text Add to dashboard Cite

Neural Network 3D Body Pose Tracking and Prediction for Motion-to-Photon Latency Compensation in Distributed Virtual Reality

Cited by 3 publications

References 19 publications

The Effects of Full-Body Avatar Movement Predictions in Virtual Reality using Neural Networks

The Effects of Full-Body Avatar Movement Predictions in Virtual Reality using Neural Networks

A Dual Perspective of Human Motion Analysis - 3D Pose Estimation and 2D Trajectory Prediction

Multi-granularity spatial temporal graph convolution network with consecutive attention for human motion prediction

Contact Info

Product

Resources

About