InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Image

Moon, Gyeongsik; Yu, Shoou-I; He, Wen; Shiratori, Takaaki; Lee, Kyoung Mu

doi:10.1007/978-3-030-58565-5_33

Cited by 190 publications

(329 citation statements)

References 42 publications

Supporting

Mentioning

327

Contrasting

Order By: Relevance

“…We believe that our dataset will allow stereo-vision research communities to promote their own studies. We also note that our dataset is still smaller in size than the datasets that have been recently introduced for RGB and RGBD sensors [8], [9], [36], [48]. Additionally, these other datasets include various gestures, such as mid-air gestures and object interaction gestures.…”

Section: ) Discussion and Limitationsmentioning

confidence: 99%

“…STB contains 18 K pairs of stereo images from real-world scenarios. Although STB is widely used as a benchmark dataset for hand pose estimation [6], [9], [34], [36], [45], it contains hand images captured from a third-person perspective, which does not align with our research goals. In this study, we collected a novel dataset using a stereo sensor from an egocentric view.…”

Section: B Datasets For Hand Pose Estimationmentioning

confidence: 99%

“…We evaluated the benefits of using both MAB and SAB modules for feature extraction. We selected AttentionNet [55] and ResNet50 [57] as comparative models because AttentionNet contains MABs and ResNet50 is the most commonly used architecture for hand pose estimation [7], [9], [34], [36]. We used the same architectures for the AttentionNet and ResNet50 as those presented in previous works [55], [57] for feature extractors.…”

Section: ) Attention-based Architecturementioning

confidence: 99%

“…We compared the proposed model, including StereoNet and top-down pipeline, against several recent hand pose estimation methods on a widely used benchmark dataset, namely, STB. We used the end-point error (EPE) matrix, defined as the mean Euclidean distance between an estimated hand pose and its ground truth, for fair evaluation as used in [13], [34]- [36], [40], [47].…”

Section: Comparison With State-of-the-art Techniquesmentioning

confidence: 99%

“…We first compare the relative hand pose estimation methods, where the input of a model is a well-cropped hand image. To train StereoNet, we followed the setup used in [36]. Table 6 lists the performance of recent hand pose estimation methods.…”

Section: ) Relative Hand Pose Estimationmentioning

confidence: 99%

See 4 more Smart Citations

Stereo Feature Learning Based on Attention and Geometry for Absolute Hand Pose Estimation in Egocentric Stereo Views

et al. 2021

View full text Add to dashboard Cite

Egocentric hand pose estimation is significant for wearable cameras since the hand interactions are captured from an egocentric viewpoint. Several studies on hand pose estimation have recently been presented based on RGBD or RGB sensors. Although these methods provide accurate hand pose estimation, they have several limitations. For example, RGB-based techniques have intrinsic difficulty in converting relative 3D poses into absolute 3D poses, and RGBD-based techniques only work in indoor environments. Recently, stereo-sensor-based techniques have gained increasing attention owing to their potential to overcome these limitations. However, to the best of our knowledge, there are few techniques and no real datasets available for egocentric stereo vision. In this paper, we propose a top-down pipeline for estimating absolute 3D hand poses using stereo sensors, as well as a novel dataset for training. Our top-down pipeline consists of two steps: hand detection and hand pose estimation. Hand detection detects hand areas and then is followed by hand pose estimation, which estimates the positions of the hand joints. In particular, for hand pose estimation with a stereo camera, we propose an attention-based architecture called StereoNet, a geometry-based loss function called StereoLoss, and a novel 2D disparity map called StereoDMap for effective stereo feature learning. To collect the dataset, we proposed a novel annotation method that helps reduce human annotation efforts. Our dataset is publicly available at https://github.com/seo0914/SEH. We conducted comprehensive experiments to demonstrate the effectiveness of our approach compared with the state-of-the-art methods.INDEX TERMS Hand pose estimation, stereo vision, wearable sensors, egocentric view.

show abstract

Section: ) Discussion and Limitationsmentioning

confidence: 99%

Section: B Datasets For Hand Pose Estimationmentioning

confidence: 99%

Section: ) Attention-based Architecturementioning

confidence: 99%

Section: Comparison With State-of-the-art Techniquesmentioning

confidence: 99%

Section: ) Relative Hand Pose Estimationmentioning

confidence: 99%

See 3 more Smart Citations

Stereo Feature Learning Based on Attention and Geometry for Absolute Hand Pose Estimation in Egocentric Stereo Views

et al. 2021

View full text Add to dashboard Cite

show abstract

HandDGCL: Two‐hand 3D reconstruction based disturbing graph contrastive learning

Han

Yao

Wang

et al. 2023

Computer Animation & Virtual

View full text Add to dashboard Cite

Virtual reality (VR) and augmented reality (AR) applications are becoming increasingly prevalent. However, constructing realistic 3D hands, especially when two hands are interacting, from a single RGB image remains a major challenge due to severe mutual occlusion and the enormous diversity of hand poses. In this article, we propose a disturbing graph contrastive learning strategy for two‐hand 3D reconstruction. This involves a graph disturbance network designed to generate graph feature pairs to enhance the consistency of the two‐hand pose features. A contrastive learning module leverages high‐quality generative features for a strong feature expression. We further propose a similarity distinguish method to divide positive and negative features for accelerating the model convergence. Additionally, a multi‐term loss is designed to balance the relation among the hand pose, the visual scale and the viewpoint position. Our model has achieved state‐of‐the‐art results in the InterHand2.6M benchmark. Ablation studies show the model's great ability to correct unreasonable hand movements. In subjective assessments, our graph disturbance learning method significantly improves the construction of realistic 3D hands, especially when two hands are interacting.

show abstract

DeepHandMesh: A Weakly-Supervised Deep Encoder-Decoder Framework for High-Fidelity Hand Mesh Modeling

Moon

Shiratori

Lee

2020

Lecture Notes in Computer Science

View full text Add to dashboard Cite

InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Image

Cited by 190 publications

References 42 publications

Stereo Feature Learning Based on Attention and Geometry for Absolute Hand Pose Estimation in Egocentric Stereo Views

Stereo Feature Learning Based on Attention and Geometry for Absolute Hand Pose Estimation in Egocentric Stereo Views

HandDGCL: Two‐hand 3D reconstruction based disturbing graph contrastive learning

DeepHandMesh: A Weakly-Supervised Deep Encoder-Decoder Framework for High-Fidelity Hand Mesh Modeling

Contact Info

Product

Resources

About