SparsePoser: Real-time Full-body Motion Reconstruction from Sparse Data

Abstract

Accurate and reliable human motion reconstruction is crucial for creating natural interactions of full-body avatars in Virtual Reality (VR) and entertainment applications. As the Metaverse and social applications gain popularity, users are seeking cost-effective solutions to create full-body animations that are comparable in quality to those produced by commercial motion capture systems. In order to provide affordable solutions though, it is important to minimize the number of sensors attached to the subject’s body. Unfortunately, reconstructing the full-body pose from sparse data is a heavily under-determined problem. Some studies that use IMU sensors face challenges in reconstructing the pose due to positional drift and ambiguity of the poses. In recent years, some mainstream VR systems have released 6-degree-of-freedom (6-DoF) tracking devices providing positional and rotational information. Nevertheless, most solutions for reconstructing full-body poses rely on traditional inverse kinematics (IK) solutions, which often produce non-continuous and unnatural poses. In this paper, we introduce SparsePoser, a novel deep learning-based solution for reconstructing a full-body pose from a reduced set of six tracking devices. Our system incorporates a convolutional-based autoencoder that synthesizes high-quality continuous human poses by learning the human motion manifold from motion capture data. Then, we employ a learned IK component, made of multiple lightweight feed-forward neural networks, to adjust the hands and feet towards the corresponding trackers. We extensively evaluate our method on publicly available motion capture datasets and with real-time live demos. We show that our method outperforms state-of-the-art techniques using IMU sensors or 6-DoF tracking devices, and can be used for users with different body dimensions and proportions.

Evaluating Visual-Spatiotemporal Co-Registration of a Physics-Based Virtual Reality Haptic Interface

“…In addition, Abdlkarim et al [51] evaluated the accuracy, delay, and overall performance of the Meta Quest 2 hand-tracking system by implementing several methods to assess positional error, delay, and finger jointangle accuracy of the system. The HTV VIVE VR system has also been utilized to develop methods for real-time full-body human tracking using the HMD and multiple VR trackers [53], [66]. These methods used solutions based on IK [66] or a combination of Deep Neural Networks and IK [53] to generate precise and dependable replication of human movement for comprehensive body avatars in VR.…”

“…The HTV VIVE VR system has also been utilized to develop methods for real-time full-body human tracking using the HMD and multiple VR trackers [53], [66]. These methods used solutions based on IK [66] or a combination of Deep Neural Networks and IK [53] to generate precise and dependable replication of human movement for comprehensive body avatars in VR. The authors also subjectively and objectively evaluated these systems in terms of user experience [66], positional and angular errors [53], and end-to-latency [66].…”

“…These methods used solutions based on IK [66] or a combination of Deep Neural Networks and IK [53] to generate precise and dependable replication of human movement for comprehensive body avatars in VR. The authors also subjectively and objectively evaluated these systems in terms of user experience [66], positional and angular errors [53], and end-to-latency [66]. Nevertheless, IK is an established technique for body tracking, which may not be suitable for other applications, such as multiple object tracking in virtual industrial training scenarios.…”

“…A few previous studies have evaluated spatial and temporal co-registration errors of VR systems [42], [50], [51], [52], [53]. However, these aforementioned studies either employed a single tracked object [53] and/or expensive thirdparty software (such as Vizard 5.5 in [42] and a software plugin in [51]) and hardware (e.g., for implementing the absolute pose estimator algorithm for calibration and tracking in [50] and an industrial serial robot as the ground truth device [52]) for calibrating and assessing the spatiotemporal registration of their respective VR systems.…”

“…A few previous studies have evaluated spatial and temporal co-registration errors of VR systems [42], [50], [51], [52], [53]. However, these aforementioned studies either employed a single tracked object [53] and/or expensive thirdparty software (such as Vizard 5.5 in [42] and a software plugin in [51]) and hardware (e.g., for implementing the absolute pose estimator algorithm for calibration and tracking in [50] and an industrial serial robot as the ground truth device [52]) for calibrating and assessing the spatiotemporal registration of their respective VR systems. To our knowledge, no previous studies have reported any empirical evidence of spatial and temporal co-registration and visual registration of a physics-based VR system simulating tasks characterized by perceptual and/or motor components (e.g., human-robot collaborative or HRC tasks) using multiple tracked objects.…”