ScrewNet: Category-Independent Articulation Model Estimation From Depth Images Using Screw Theory

Jain, Ajinkya; Lioutikov, Rudolf; Chuck, Caleb; Niekum, Scott

doi:10.48550/arxiv.2008.10518

Cited by 11 publications

(26 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On the other hand, methods based on passive visual observations formulate the articulation parameters inference problem as a regression task [15]- [17] and leverage large synthetic datasets [18], [19]. Abbatematteo et al [15] map a segmented depth image of an object into its kinematic model parameters and geometry by using mixture density networks.…”

Section: A Learning Articulated Modelsmentioning

confidence: 99%

Articulated Object Interaction in Unknown Scenes with Whole-Body Mobile Manipulation

Mittal¹,

Hoeller²,

Farshidian³

et al. 2021

Preprint

View full text Add to dashboard Cite

A kitchen assistant needs to operate human-scale objects, such as cabinets and ovens, in unmapped environments with dynamic obstacles. Autonomous interactions in such realworld environments require integrating dexterous manipulation and fluid mobility. While mobile manipulators in different form-factors provide an extended workspace, their real-world adoption has been limited. This limitation is in part due to two main reasons: 1) inability to interact with unknown humanscale objects such as cabinets and ovens, and 2) inefficient coordination between the arm and the mobile base. Executing a high-level task for general objects requires a perceptual understanding of the object as well as adaptive whole-body control among dynamic obstacles. In this paper, we propose a two-stage architecture for autonomous interaction with large articulated objects in unknown environments. The first stage uses a learned model to estimate the articulated model of a target object from an RGB-D input and predicts an actionconditional sequence of states for interaction. The second stage comprises of a whole-body motion controller to manipulate the object along the generated kinematic plan. We show that our proposed pipeline can handle complicated static and dynamic kitchen settings. Moreover, we demonstrate that the proposed approach achieves better performance than commonly used control methods in mobile manipulation. For additional material, please check: www.pair.toronto.edu/articulated-mm/.

show abstract

Section: A Learning Articulated Modelsmentioning

confidence: 99%

Articulated Object Interaction in Unknown Scenes with Whole-Body Mobile Manipulation

Mittal¹,

Hoeller²,

Farshidian³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Synthetic datasets [30,33] are used to train regression-based models to directly predict part motion parameters. However, methods train on these dataset require accurate depth information [7,9,18,34]. Thus it is not straightforward for one to directly apply them to real world scenes due to noisy depth estimation and domain gap differences.…”

Section: D Object Pose and Motion Estimationmentioning

confidence: 99%

D3D-HOI: Dynamic 3D Human-Object Interactions from Videos

Xu,

Joo,

Mori

et al. 2021

Preprint

View full text Add to dashboard Cite

We address the problem of recovering dynamic 3D human-object interactions given a video input. Our focus is on reconstructing the articulating 3D object during manipulation. To enable research in this direction, we collect a dataset of interaction videos with common objects (e.g., laptop, fridge, dishwasher). We annotate the 3D object pose, shape, articulation state, and estimate the 3D mesh of the manipulator using the approach from Joo et al. [10]. Above are rendered frames from the ground truth annotations.

show abstract

“…Previous works [42,43,44,45] have also explored various robotic planning and control methods for manipulating 3D articulated objects. More recent works further leveraged learning techniques for better predicting articulated part configurations, parameters, and states [6,7,4,46,3,5,47], estimating kinematic structures [1,2], as well as manipulating 3D articulated objects with the learned visual knowledge [8,9,10,11,12]. While most of these works represented visual data with link poses, joint parameters, and kinematic structures, such standardized abstractions may be insufficient if fine-grained part geometry, such as drawer handles and faucet switches that exhibit rich geometric diversity among different shapes, matters for downstream robotic tasks and motion planning.…”

Section: Related Workmentioning

confidence: 99%

“…There has been a long line of research studying the perception and manipulation of 3D articulated objects in computer vision and robotics. On the perception side, researchers have developed various successful visual systems for estimating kinematic structures [1,2], articulated part poses [3,4,5], and Figure 1: Given an input 3D articulated object (a), we propose a novel perception-interaction handshaking point for robotic manipulation tasks -object-centric actionable visual priors, including per-point visual action affordance predictions (b) indicating where to interact, and diverse trajectory proposals (c) for selected contact points (marked with green dots) suggesting how to interact.…”

Section: Introductionmentioning

confidence: 99%

VAT-Mart: Learning Visual Action Trajectory Proposals for Manipulating 3D ARTiculated Objects

Wu¹,

Zhao²,

Mo³

et al. 2021

Preprint

View full text Add to dashboard Cite

Perceiving and manipulating 3D articulated objects (e.g., cabinets, doors) in human environments is an important yet challenging task for future home-assistant robots. The space of 3D articulated objects is exceptionally rich in their myriad semantic categories, diverse shape geometry, and complicated part functionality. Previous works mostly abstract kinematic structure with estimated joint parameters and part poses as the visual representations for manipulating 3D articulated objects. In this paper, we propose object-centric actionable visual priors as a novel perception-interaction handshaking point that the perception system outputs more actionable guidance than kinematic structure estimation, by predicting dense geometry-aware, interaction-aware, and task-aware visual action affordance and trajectory proposals. We design an interaction-for-perception framework VAT-MART to learn such actionable visual representations by simultaneously training a curiosity-driven reinforcement learning policy exploring diverse interaction trajectories and a perception module summarizing and generalizing the explored knowledge for pointwise predictions among diverse shapes. Experiments prove the effectiveness of the proposed approach using the large-scale PartNet-Mobility dataset in SAPIEN environment and show promising generalization capabilities to novel test shapes, unseen object categories, and real-world data. Please check the project webpage for code, data, video, and more materials.

show abstract

ScrewNet: Category-Independent Articulation Model Estimation From Depth Images Using Screw Theory

Cited by 11 publications

References 30 publications

Articulated Object Interaction in Unknown Scenes with Whole-Body Mobile Manipulation

Articulated Object Interaction in Unknown Scenes with Whole-Body Mobile Manipulation

D3D-HOI: Dynamic 3D Human-Object Interactions from Videos

VAT-Mart: Learning Visual Action Trajectory Proposals for Manipulating 3D ARTiculated Objects

Contact Info

Product

Resources

About