Pose-Guided Hierarchical Graph Reasoning for 3-D Hand Pose Estimation From a Single Depth Image

Ren, Pengfei; Sun, Haifeng; Hao, Jiachang; Qi, Qi; Wang, Jingyu; Liao, Jianxin

doi:10.1109/tcyb.2021.3083637

Cited by 11 publications

(6 citation statements)

References 78 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We compare HandGCNFormer with various existing methods (Ge et al, 2018a,b;Moon et al, 2018;Wan et al, 2018;Du et al, 2019;Ren et al, 2019Ren et al, , 2021aXiong et al, 2019;Chen et al, 2020;Huang et al, 2020c) across standard NYU, ICVL, MSRA, and Hands2017 benchmarks. Table 4 presents the comparison results using the mean 3D distance error metric.…”

Section: Results Of D Hand Pose Estimationmentioning

confidence: 99%

“…However, because the post-processing step of extracting joint coordinates from heatmaps is similar across these methods, they often cannot be trained end-to-end. Subsequently, hybrid approaches (Sun et al, 2018;Huang et al, 2020c;Malik et al, 2020;Ren et al, 2021a) emerged, combining the strengths of both regression-based and detection-based methods. For example, AWR (Huang et al, 2020c) transforms 3D hand joint coordinates into a 3D heatmap and unit vector field, providing direct supervision for joint position.…”

Section: Hand Pose Estimationmentioning

confidence: 99%

“…These approaches for 2D-to-3D lifting underscore the significance of topology information in mitigating depth ambiguity. PHG (Ren et al, 2021a) endeavors to establish longrange dependencies among hand joints by leveraging a cascaded GCN module, achieving state-of-the-art performance. However, the cascaded GCN module tends to introduce noisy information from extended neighbor nodes exponentially while constructing global relationships, resulting in over-smoothing of the model.…”

Section: Graph Convolutional Networkmentioning

confidence: 99%

“…Hand pose estimation seeks to ascertain the spatial coordinates of hand joints from either a single depth image or an RGB image. Depth-based methods have made impressive progress as commodity depth cameras become cheaper and more accurate (Guo et al, 2017b;Ren et al, 2019Ren et al, , 2021aXiong et al, 2019;Chen et al, 2020;Fang et al, 2020;Huang et al, 2020a,c). However, in situations where there is significant self-occlusion and high selfsimilarity in hand poses, it remains highly challenging, as illustrated in Figures 1A, B.…”

Section: Introductionmentioning

confidence: 99%

“…Previous studies (Zhao et al, 2019;Bai et al, 2021;Tunga et al, 2021) have demonstrated the capability of graph convolutional networks (GCNs) to represent such topological relationships. Recently, the pose-guided hierarchical graph convolution (PHG) method (Ren et al, 2021a) attempts to model the long-range dependencies between hand components by stacking multiple GCN layers. However, the cascaded GCNs can lead to error accumulation in long-term graph elements and worsen the problem of over-smoothing.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

3D hand pose and mesh estimation via a generic Topology-aware Transformer model

Yu,

Wang,

Chen

et al. 2024

Front. Neurorobot.

View full text Add to dashboard Cite

In Human-Robot Interaction (HRI), accurate 3D hand pose and mesh estimation hold critical importance. However, inferring reasonable and accurate poses in severe self-occlusion and high self-similarity remains an inherent challenge. In order to alleviate the ambiguity caused by invisible and similar joints during HRI, we propose a new Topology-aware Transformer network named HandGCNFormer with depth image as input, incorporating prior knowledge of hand kinematic topology into the network while modeling long-range contextual information. Specifically, we propose a novel Graphformer decoder with an additional Node-offset Graph Convolutional layer (NoffGConv). The Graphformer decoder optimizes the synergy between the Transformer and GCN, capturing long-range dependencies and local topological connections between joints. On top of that, we replace the standard MLP prediction head with a novel Topology-aware head to better exploit local topological constraints for more reasonable and accurate poses. Our method achieves state-of-the-art 3D hand pose estimation performance on four challenging datasets, including Hands2017, NYU, ICVL, and MSRA. To further demonstrate the effectiveness and scalability of our proposed Graphformer Decoder and Topology aware head, we extend our framework to HandGCNFormer-Mesh for the 3D hand mesh estimation task. The extended framework efficiently integrates a shape regressor with the original Graphformer Decoder and Topology aware head, producing Mano parameters. The results on the HO-3D dataset, which contains various and challenging occlusions, show that our HandGCNFormer-Mesh achieves competitive results compared to previous state-of-the-art 3D hand mesh estimation methods.

show abstract