TravelNet: Self-supervised Physically Plausible Hand Motion Learning from Monocular Color Images

Zhao, Zimeng; Zhao, Xibin; Wang, Yangang

doi:10.1109/iccv48922.2021.01146

Cited by 12 publications

(4 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To propagate temporal information across frames, one approach suggested in a study (Hossain and Little, 2018) involves employing a sequence-to-sequence model, while another study introduces the use of a spatial-temporal graph. Video input is also utilized to exploit temporal features , Chen et al, 2021a, Kocabas et al, 2020, Ye et al, 2023, Zhao et al, 2021, but in this case, a frame-to-frame real-time operation is not feasible due to the heavy structural system or the necessity of future information. Han et al (Han et al, 2020(Han et al, , 2022) demonstrate an integrated handtracking system within the Oculus Quest VR headset, utilizing the available inputs that depend on the hardware and proposed a regression network using tracking history.…”

Section: Temporal Coherence On Pose Estimationmentioning

confidence: 99%

Temporally Enhanced Graph Convolutional Network for Hand Tracking from an Egocentric Camera

Cho,

Ha,

Jeon

et al. 2024

Preprint

View full text Add to dashboard Cite

We propose a robust 3D hand tracking system in various hand action environments, including hand-object interaction, which utilizes a single color image and a previous pose prediction as input. We observe that existing methods deterministically exploit temporal information in motion space, failing to address realistic diverse hand motions. Also, prior methods paid less attention to efficiency as well as robust performance, i.e., the balance issues between time and accuracy. The Temporally Enhanced Graph Convolutional Network (TE-GCN) utilizes a 2-stage framework to encode temporal information adaptively. The system establishes balance by adopting an adaptive GCN, which effectively learns the spatial dependency between hand mesh vertices. Furthermore, the system leverages the previous prediction by estimating the relevance across image features through the attention mechanism. The proposed method achieves state-of-the-art balanced performance on challenging benchmarks and demonstrates robust results on various hand motions in real scenes. Moreover, the hand tracking system is integrated into a recent HMD with an off-loading framework, achieving a real-time framerate while maintaining high performance. Our study improves the usability of a high-performance hand-tracking method, which can be generalized to other algorithms and contributes to the usage of HMD in everyday life. Our code with the HMD project will be available at https://github.com/UVR-WJCHO/TEGCN_on_Hololens2

show abstract

Section: Temporal Coherence On Pose Estimationmentioning

confidence: 99%

Temporally Enhanced Graph Convolutional Network for Hand Tracking from an Egocentric Camera

Cho,

Ha,

Jeon

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…For RGB-based methods, most works formulate hand reconstruction as a problem of regressing the MANO coefficients (Zhang et al 2019;Zhou et al 2020b;Yang et al 2020b;Hasson et al 2019;Zhang et al 2021b;Boukhayma, Bem, and Torr 2019;Baek, Kim, and Kim 2019;Zimmermann et al 2019;Chen et al 2021c;Zhao, Zhao, and Wang 2021;Cao et al 2021;Baek, Kim, and Kim 2020;Jiang et al 2021;Liu et al 2021;Zhang et al 2021a;Tse et al 2022). Other widely-used representations include voxel-based (Iqbal et al 2018;Yang, Chen, and Yao 2021), implicit-functionbased (Mescheder et al 2019), and vertex-based (Ge et al 2019;Kulon et al 2020;Chen et al 2021b;Lin, Wang, and Liu 2021a,b).…”

Section: Related Workmentioning

confidence: 99%

SiMA-Hand: Boosting 3D Hand-Mesh Reconstruction by Single-to-Multi-View Adaptation

Wang,

Xu,

Heng

et al. 2024

AAAI

View full text Add to dashboard Cite

Estimating 3D hand mesh from RGB images is a longstanding track, in which occlusion is one of the most challenging problems. Existing attempts towards this task often fail when the occlusion dominates the image space. In this paper, we propose SiMA-Hand, aiming to boost the mesh reconstruction performance by Single-to-Multi-view Adaptation. First, we design a multi-view hand reconstructor to fuse information across multiple views by holistically adopting feature fusion at image, joint, and vertex levels. Then, we introduce a single-view hand reconstructor equipped with SiMA. Though taking only one view as input at inference, the shape and orientation features in the single-view reconstructor can be enriched by learning non-occluded knowledge from the extra views at training, enhancing the reconstruction precision on the occluded regions. We conduct experiments on the Dex-YCB and HanCo benchmarks with challenging object- and self-caused occlusion cases, manifesting that SiMA-Hand consistently achieves superior performance over the state of the arts. Code will be released on https://github.com/JoyboyWang/SiMA-Hand Pytorch.

show abstract

“…To facilitate the sampling, the hand pose θ represented by the axis-angles is converted into ϑ represented by the Euler angles. Two schemes are adopted for local DoF: retain all 45 or only 20 physically plausible ones [67,70], i.e. |ϑ| = 48 or 23.…”

Section: Physical Contact Recoverymentioning

confidence: 99%

Stability-driven Contact Reconstruction From Monocular Color Images

Zhao¹,

Zuo²,

Xie³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Physical contact provides additional constraints for hand-object state reconstruction as well as a basis for further understanding of interaction affordances. Estimating these severely occluded regions from monocular images presents a considerable challenge. Existing methods optimize the hand-object contact driven by distance threshold or prior from contact-labeled datasets. However, due to the number of subjects and objects involved in these indoor datasets being limited, the learned contact patterns could not be generalized easily. Our key idea is to reconstruct the contact pattern directly from monocular images, and then utilize the physical stability criterion in the simulation to optimize it. This criterion is defined by the resultant forces and contact distribution computed by the physics engine. Compared to existing solutions, our framework can be adapted to more personalized hands and diverse object shapes. Furthermore, an interaction dataset with extra physical attributes is created to verify the sim-to-real consistency of our methods. Through comprehensive evaluations, hand-object contact can be reconstructed with both accuracy and stability by the proposed framework.

show abstract

TravelNet: Self-supervised Physically Plausible Hand Motion Learning from Monocular Color Images

Cited by 12 publications

References 39 publications

Temporally Enhanced Graph Convolutional Network for Hand Tracking from an Egocentric Camera

Temporally Enhanced Graph Convolutional Network for Hand Tracking from an Egocentric Camera

SiMA-Hand: Boosting 3D Hand-Mesh Reconstruction by Single-to-Multi-View Adaptation

Stability-driven Contact Reconstruction From Monocular Color Images

Contact Info

Product

Resources

About