ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation

Fan, Zicong; Taheri, Omid; Tzionas, Dimitrios; Kocabas, Muhammed; Kaufmann, Manuel; Black, Michael J.; Hilliges, Otmar

doi:10.1109/cvpr52729.2023.01244

Cited by 62 publications

(4 citation statements)

References 74 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Each subject performed 80 trials, ensuring 300 interactions for each of the 4 objects. Such participants’ numerosity is generally higher than that of other object manipulation datasets in the literature, as highlighted by the review of Huang et al (2016) and, as recent examples, the works by Kratzer et al (2021); Xompero et al (2022) and Fan et al (2023).…”

Section: Methodsmentioning

confidence: 92%

The effects of selected object features on a pick-and-place task: A human multimodal dataset

Lastrico,

Belcamino,

Carfì

et al. 2023

The International Journal of Robotics Research

View full text Add to dashboard Cite

We propose a dataset to study the influence of object-specific characteristics on human pick-and-place movements and compare the quality of the motion kinematics extracted by various sensors. This dataset is also suitable for promoting a broader discussion on general learning problems in the hand-object interaction domain, such as intention recognition or motion generation with applications in the Robotics field. The dataset consists of the recordings of 15 subjects performing 80 repetitions of a pick-and-place action under various experimental conditions, for a total of 1200 pick-and-places. The data has been collected thanks to a multimodal setup composed of multiple cameras, observing the actions from different perspectives, a motion capture system, and a wrist-worn inertial measurement unit. All the objects manipulated in the experiments are identical in shape, size, and appearance but differ in weight and liquid filling, which influences the carefulness required for their handling.

show abstract

Section: Methodsmentioning

confidence: 92%

The effects of selected object features on a pick-and-place task: A human multimodal dataset

Lastrico,

Belcamino,

Carfì

et al. 2023

The International Journal of Robotics Research

View full text Add to dashboard Cite

show abstract

“…Hand-object interaction is important in virtual reality and robotics. Recently, with the advent of datasets that con-tain both hand and object annotations (Hampali et al 2020;Taheri et al 2020;Hasson et al 2019;Fan et al 2023), impressive progress has been made. Early methods mainly focus on modeling static grasping interaction.…”

Section: Related Work Hand Object Interactionmentioning

confidence: 99%

“…Some methods (Hasson et al 2021(Hasson et al , 2020Chen et al 2023) can estimate/reconstruct interactions from video input. With the hand-object interaction motion datasets (Taheri et al 2020;Hampali et al 2020;Liu et al 2022b;Fan et al 2023), interaction motions can be generated (Zhang et al 2021;Taheri et al 2022;Wu et al 2022;Zheng et al 2023). As the first method to address refining perturbed 3D hand-object interaction sequence, Zhou et al (Zhou et al 2022) proposed an objectcentric representation with a spatial-temporal modeling architecture.…”

Section: Related Work Hand Object Interactionmentioning

confidence: 99%

Hand-Centric Motion Refinement for 3D Hand-Object Interaction via Hierarchical Spatial-Temporal Modeling

Hao,

Zhang,

Zhuo

et al. 2024

AAAI

View full text Add to dashboard Cite

Hands are the main medium when people interact with the world. Generating proper 3D motion for hand-object interaction is vital for applications such as virtual reality and robotics. Although grasp tracking or object manipulation synthesis can produce coarse hand motion, this kind of motion is inevitably noisy and full of jitter. To address this problem, we propose a data-driven method for coarse motion refinement. First, we design a hand-centric representation to describe the dynamic spatial-temporal relation between hands and objects. Compared to the object-centric representation, our hand-centric representation is straightforward and does not require an ambiguous projection process that converts object-based prediction into hand motion. Second, to capture the dynamic clues of hand-object interaction, we propose a new architecture that models the spatial and temporal structure in a hierarchical manner. Extensive experiments demonstrate that our method outperforms previous methods by a noticeable margin.

show abstract

“…These constraints can indeed limit the applicability of diffusion models. Currently, various human motion datasets exist, coupled with diverse conditions, including text [GZZ*22, GZW*20], music [LYRK21], audio [FM18], video [IPOS14, MRC*17], scene descriptions [ALV*23, GM‐SPM21,ZYM*22,YK19,ZMZ*22] and objects [TGBT20,FTT*23, BXP*22, LWL23]. Continuing to develop and share such large‐scale, diverse, and high‐quality paired motion datasets is a valuable investment in the relevant research and development communities.…”

Section: Towards 4d Spatio‐temporal Diffusionmentioning

confidence: 99%

State of the Art on Diffusion Models for Visual Computing

Po,

Yifan,

Golyanik

et al. 2024

Computer Graphics Forum

View full text Add to dashboard Cite

The field of visual computing is rapidly advancing due to the emergence of generative artificial intelligence (AI), which unlocks unprecedented capabilities for the generation, editing, and reconstruction of images, videos, and 3D scenes. In these domains, diffusion models are the generative AI architecture of choice. Within the last year alone, the literature on diffusion‐based tools and applications has seen exponential growth and relevant papers are published across the computer graphics, computer vision, and AI communities with new works appearing daily on arXiv. This rapid growth of the field makes it difficult to keep up with all recent developments. The goal of this state‐of‐the‐art report (STAR) is to introduce the basic mathematical concepts of diffusion models, implementation details and design choices of the popular Stable Diffusion model, as well as overview important aspects of these generative AI tools, including personalization, conditioning, inversion, among others. Moreover, we give a comprehensive overview of the rapidly growing literature on diffusion‐based generation and editing, categorized by the type of generated medium, including 2D images, videos, 3D objects, locomotion, and 4D scenes. Finally, we discuss available datasets, metrics, open challenges, and social implications. This STAR provides an intuitive starting point to explore this exciting topic for researchers, artists, and practitioners alike.

show abstract

ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation

Cited by 62 publications

References 74 publications

The effects of selected object features on a pick-and-place task: A human multimodal dataset

The effects of selected object features on a pick-and-place task: A human multimodal dataset

Hand-Centric Motion Refinement for 3D Hand-Object Interaction via Hierarchical Spatial-Temporal Modeling

State of the Art on Diffusion Models for Visual Computing

Contact Info

Product

Resources

About