Rotating without Seeing: Towards In-hand Dexterity through Touch

Yin, Zhao-Heng; Huang, Binghao; Qin, Yuzhe; Chen, Qifeng; Wang, Xiaolong

doi:10.15607/rss.2023.xix.036

Cited by 34 publications

(2 citation statements)

References 61 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this work, we construct a surface prior using contacts between hands and objects and simplify the learning problem. Actually, some works in robotics (Bicchi and Kumar 2000;Li et al 2020;Buescher et al 2015;Yin et al 2023;Tse et al 2022b) have shown that contacts could provide rich cues about the object shape and how to manipulate the given object. Some recent systems (Yin et al 2023;Jain et al 2019) can successfully manipulate different objects by using contact sensors.…”

Section: Related Workmentioning

confidence: 99%

“…Actually, some works in robotics (Bicchi and Kumar 2000;Li et al 2020;Buescher et al 2015;Yin et al 2023;Tse et al 2022b) have shown that contacts could provide rich cues about the object shape and how to manipulate the given object. Some recent systems (Yin et al 2023;Jain et al 2019) can successfully manipulate different objects by using contact sensors. However, how to use contacts to benefit hand-held reconstruction is under-explored in our task.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Learning Explicit Contact for Implicit Reconstruction of Hand-Held Objects from Monocular Images

Hu,

Zhang,

Chen

et al. 2024

AAAI

View full text Add to dashboard Cite

Reconstructing hand-held objects from monocular RGB images is an appealing yet challenging task. In this task, contacts between hands and objects provide important cues for recovering the 3D geometry of the hand-held objects. Though recent works have employed implicit functions to achieve impressive progress, they ignore formulating contacts in their frameworks, which results in producing less realistic object meshes. In this work, we explore how to model contacts in an explicit way to benefit the implicit reconstruction of hand-held objects. Our method consists of two components: explicit contact prediction and implicit shape reconstruction. In the first part, we propose a new subtask of directly estimating 3D hand-object contacts from a single image. The part-level and vertex-level graph-based transformers are cascaded and jointly learned in a coarse-to-fine manner for more accurate contact probabilities. In the second part, we introduce a novel method to diffuse estimated contact states from the hand mesh surface to nearby 3D space and leverage diffused contact probabilities to construct the implicit neural representation for the manipulated object. Benefiting from estimating the interaction patterns between the hand and the object, our method can reconstruct more realistic object meshes, especially for object parts that are in contact with hands. Extensive experiments on challenging benchmarks show that the proposed method outperforms the current state of the arts by a great margin. Our code is publicly available at https://junxinghu.github.io/projects/hoi.html.

show abstract