HOnnotate: A Method for 3D Annotation of Hand and Object Poses

Hampali, Shreyas; Rad, Mahdi; Oberweger, Markus; Lepetit, Vincent

doi:10.1109/cvpr42600.2020.00326

Cited by 299 publications

(308 citation statements)

References 54 publications

Supporting

Mentioning

308

Contrasting

Order By: Relevance

“…Due to the lack of large scale 3D ground truth data, synthetic data has often been used for training [14,29,49]. Recently, instead of estimating the hand skeleton, recovering the pose and the surface of the hand has become popular using statistical hand models, e.g., the MANO model [74], that can represent a variety of hand shapes and poses [25,93,97]. Using the template derived from MANO, [41] show that it is also possible to regress hand meshes directly using mesh convolution.…”

Section: Related Workmentioning

confidence: 99%

“…Hand-object interaction is generated using a physics simulator, GraspIt [53], resulting in high-quality hand-object interaction. Due to the limited number of grasp types in the FHB dataset [19] and the HO-3D dataset [25], they are not suitable for training the generative model (see Appendix B). Instead, we use them to test the generalization ability of the generative model trained on the ObMan grasps.…”

Section: Datasetmentioning

confidence: 99%

“…In this section, we provide detailed analyses on the FHB [19] and the HO-3D [25] datasets. Although these datasets considerably contribute to the studies of hand-object interactions with detailed 3D annotation, our analyse shows that they might not be suitable for learning human grasps and modelling the accurate contact relation between hand and object.…”

Section: B Dataset Analysismentioning

confidence: 99%

See 2 more Smart Citations

Grasping Field: Learning Implicit Representations for Human Grasps

Karunratanakul

Yang

Zhang

et al. 2020

2020 International Conference on 3D Vision (3DV)

168

122

View full text Add to dashboard Cite

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Datasetmentioning

confidence: 99%

Section: B Dataset Analysismentioning

confidence: 99%

See 1 more Smart Citation

Grasping Field: Learning Implicit Representations for Human Grasps

Karunratanakul

Yang

Zhang

et al. 2020

2020 International Conference on 3D Vision (3DV)

168

122

View full text Add to dashboard Cite

show abstract

“…We acquired RGB and depth frames and reconstruct the colorized overlapping point clouds for ground truth labeling while handling a real Colibri II drill. We choose this marker-less tracking approach over marker-based approaches to recover the ground truth hand and tool poses, since any markers attached to the tool or hand would be visible in the captured images and can introduce a bias for learning based methods [13].…”

Section: Real Data Generationmentioning

confidence: 99%

Towards markerless surgical tool and hand pose estimation

et al. 2021

View full text Add to dashboard Cite

Purpose: Tracking of tools and surgical activity is becoming more and more important in the context of computer assisted surgery. In this work, we present a data generation framework, dataset and baseline methods to facilitate further research in the direction of markerless hand and instrument pose estimation in realistic surgical scenarios. Methods: We developed a rendering pipeline to create inexpensive and realistic synthetic data for model pretraining. Subsequently, we propose a pipeline to capture and label real data with hand and object pose ground truth in an experimental setup to gather high-quality real data. We furthermore present three state-of-the-art RGB-based pose estimation baselines. Results: We evaluate three baseline models on the proposed datasets. The best performing baseline achieves an average tool 3D vertex error of 16.7 mm on synthetic data as well as 13.8 mm on real data which is comparable to the state-of-the art in RGB-based hand/object pose estimation. Conclusion: To the best of our knowledge, we propose the first synthetic and real data generation pipelines to generate hand and object pose labels for open surgery. We present three baseline models for RGB based object and object/hand pose estimation based on RGB frames. Our realistic synthetic data generation pipeline may contribute to overcome the data bottleneck in the surgical domain and can easily be transferred to other medical applications.

show abstract

“…Since there is no other widely acknowledged benchmark dataset for depth data of hands interacting with objects from egocentric view, we chose other standard benchmark datasets: NYU, HO3D, and HANDS2017 to test the performance of our solution on data that are not exclusively egocentric. HO3D [68] is a dataset of RGB-D images of hands manipulating with objects from a 3 rd person view. Annotations of the pose of the hand and the object are available.…”

Section: ) Other Datasetsmentioning

confidence: 99%