DexYCB: A Benchmark for Capturing Hand Grasping of Objects

Chao, Yu-Wei; Yang, Wei; Yu, Xi; Molchanov, Pavlo; Handa, Ankur; Tremblay, Jonathan; Narang, Yashraj; Wyk, Karl Van; Iqbal, Umar; Birchfield, Stan; Kautz, Jan; Fox, Dieter

doi:10.1109/cvpr46437.2021.00893

Cited by 170 publications

(113 citation statements)

References 34 publications

Supporting

Mentioning

113

Contrasting

Order By: Relevance

“…We compare the accuracy of Dexpilot-Single's online optimization with our neural network retargeter that relies on offline optimization during training. We gather a test set of 500 sequences from the DexYCB video dataset [13], which contains videos with annotated ground-truth human hand poses. For each video, at each timestep, the poses are fed to our neural network and Dexpilot-Single with a (generous) time budget of 40ms to solve.…”

Section: A Accuracy Of Retargeter Networkmentioning

confidence: 99%

Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on Youtube

Sivakumar¹,

Shaw²,

Pathak³

2022

Preprint

View full text Add to dashboard Cite

We build a system that enables any human to control a robot hand and arm, simply by demonstrating motions with their own hand. The robot observes the human operator via a single RGB camera and imitates their actions in real-time. Human hands and robot hands differ in shape, size, and joint structure, and performing this translation from a single uncalibrated camera is a highly underconstrained problem. Moreover, the retargeted trajectories must effectively execute tasks on a physical robot, which requires them to be temporally smooth and free of self-collisions. Our key insight is that while paired humanrobot correspondence data is expensive to collect, the internet contains a massive corpus of rich and diverse human hand videos. We leverage this data to train a system that understands human hands and retargets a human video stream into a robot hand-arm trajectory that is smooth, swift, safe, and semantically similar to the guiding demonstration. We demonstrate that it enables previously untrained people to teleoperate a robot on various dexterous manipulation tasks. Our low-cost, glove-free, marker-free remote teleoperation system makes robot teaching more accessible and we hope that it can aid robots that learn to act autonomously in the real world. The videos can be found at: https://robotic-telekinesis.github.io/

show abstract

Section: A Accuracy Of Retargeter Networkmentioning

confidence: 99%

Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on Youtube

Sivakumar¹,

Shaw²,

Pathak³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Not surprisingly, hand-object interaction has received much attention. This growth is accelerated by the introduction of datasets that contain both hand and object annotations [1,2,5,9,14,24,39]. Leveraging this data, a large number of methods attempt to estimate grasp parameters, such as the hand and object pose, directly from RGB images [4,10,15,16,22,26,40,42].…”

Section: Related Workmentioning

confidence: 99%

“…The grasping policy's purpose is to establish and maintain a stable grasp, whereas the motion synthesis module generates a motion to move the object to a user-specified target position. To guide the low-level grasping policy, we require a single grasp label corresponding to a static hand pose, which can be obtained either from a hand-grasping dataset [5,14] or from a state-of-the-art grasp synthesis method [19]. Crucially, we propose a reward function that is parameterized by the grasp label to incentivize the fingers to reach contact points on the object, leading to human-like grasps.…”

Section: Introductionmentioning

confidence: 99%

D-Grasp: Physically Plausible Dynamic Grasp Synthesis for Hand-Object Interactions

Christen¹,

Kocabas²,

Aksan³

et al. 2021

Preprint

View full text Add to dashboard Cite

We introduce the dynamic grasp synthesis task: given an object with a known 6D pose and a grasp reference, our goal is to generate motions that move the object to a target 6D pose. This is challenging, because it requires reasoning about the complex articulation of the human hand and the intricate physical interaction with the object. We propose a novel method that frames this problem in the reinforcement learning framework and leverages a physics simulation, both to learn and to evaluate such dynamic interactions. A hierarchical approach decomposes the task into low-level grasping and high-level motion synthesis. It can be used to generate novel hand sequences that approach, grasp, and move an object to a desired location, while retaining human-likeness. We show that our approach leads to stable grasps and generates a wide range of motions. Furthermore, even imperfect labels can be corrected by our method to generate dynamic interaction sequences. Video is available at https://eth-ait.github.io/d-grasp/.

show abstract

“…Size Mesh UP MV STB [84] real 36K × × × RHD [92] synthetic 44K × × × GANerated Hands [60] synthetic 331K × × × SeqHAND [78] synthetic 410K × × EgoDexter [61] real 3K × × × Dexter+Object [70] real 3K × × × FreiHAND [93] real 134K × × YoutubeHand [46] real 47K × × ObMan [29] synthetic 153K × × HO3D [25] real 77K × × DexYCB [8] real 528K × × H2O [47] synthetic 571K × × FPHA [16] synthetic 105K × × × H3D [87] real 22K × (15) MHP [19] real 80K × × (4) MVHM [9] synthetic 320K × (8) InterHand2.6M [59] real 2.6M × (80) ours synthetic 328K (216) Table 7. Comparison among RGB-based 3D hand datasets.…”

Section: Dataset Typementioning

confidence: 99%

“…Motivation. As shown in Table 7, many datasets are developed for 3D hand pose estimation [84,92,61,70,93,46,19,87,8,47,16,78,60]. To collect real-world hand data, existing datasets are usually captured using a multiview studio and annotated via semi-automatic model fitting [93,19].…”

Section: Synthetic Datasetmentioning

confidence: 99%

MobRecon: Mobile-Friendly Hand Mesh Reconstruction from Monocular Image

Chen¹,

Liu²,

Dong³

et al. 2021

Preprint

View full text Add to dashboard Cite

In this work, we propose a framework for singleview hand mesh reconstruction, which can simultaneously achieve high reconstruction accuracy, fast inference speed, and temporal coherence. Specifically, for 2D encoding, we propose lightweight yet effective stacked structures. Regarding 3D decoding, we provide an efficient graph operator, namely depth-separable spiral convolution. Moreover, we present a novel feature lifting module for bridging the gap between 2D and 3D representations. This module starts with a map-based position regression (MapReg) block to integrate the merits of both heatmap encoding and position regression paradigms to improve 2D accuracy and temporal coherence. Furthermore, MapReg is followed by pose pooling and pose-to-vertex lifting approaches, which transform 2D pose encodings to semantic features of 3D vertices. Overall, our hand reconstruction framework, called MobRecon, comprises affordable computational costs and miniature model size, which reaches a high inference speed of 83FPS on Apple A14 CPU. Extensive experiments on popular datasets such as FreiHAND, RHD, and HO3Dv2 demonstrate that our MobRecon achieves superior performance on reconstruction accuracy and temporal coherence. Our code is publicly available at https://github. com/SeanChenxy/HandMesh.

show abstract

DexYCB: A Benchmark for Capturing Hand Grasping of Objects

Cited by 170 publications

References 34 publications

Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on Youtube

Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on Youtube

D-Grasp: Physically Plausible Dynamic Grasp Synthesis for Hand-Object Interactions

MobRecon: Mobile-Friendly Hand Mesh Reconstruction from Monocular Image

Contact Info

Product

Resources

About