Single-view robot pose and joint angle estimation via render &amp; compare

Labbé, Yann; Carpentier, Justin; Aubry, Mathieu; Šivic, Josef

doi:10.1109/cvpr46437.2021.00170

Cited by 36 publications

(60 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Modeling the pose and shape of 3D objects from monocular images is one of the longest standing objectives of computer vision [42,52]. Recent methods train deep neural network models to compute the object shape [11,16,36,55,70] and pose [31,32,34,68] directly from image pixels. Learned object shape reconstruction from single view images has initially focused on point-cloud [48], mesh [16,66] and voxel [12,51] representations.…”

Section: Related Workmentioning

confidence: 99%

AlignSDF: Pose-Aligned Signed Distance Fields for Hand-Object Reconstruction

Chen¹,

Hasson²,

Schmid³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recent work achieved impressive progress towards joint reconstruction of hands and manipulated objects from monocular color images. Existing methods focus on two alternative representations in terms of either parametric meshes or signed distance fields (SDFs). On one side, parametric models can benefit from prior knowledge at the cost of limited shape deformations and mesh resolutions. Mesh models, hence, may fail to precisely reconstruct details such as contact surfaces of hands and objects. SDF-based methods, on the other side, can represent arbitrary details but are lacking explicit priors. In this work we aim to improve SDF models using priors provided by parametric representations. In particular, we propose a joint learning framework that disentangles the pose and the shape. We obtain hand and object poses from parametric models and use them to align SDFs in 3D space. We show that such aligned SDFs better focus on reconstructing shape details and improve reconstruction accuracy both for hands and objects. We evaluate our method and demonstrate significant improvements over the state of the art on the challenging ObMan and DexYCB benchmarks.

show abstract

Section: Related Workmentioning

confidence: 99%

AlignSDF: Pose-Aligned Signed Distance Fields for Hand-Object Reconstruction

Chen¹,

Hasson²,

Schmid³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Instead of depth maps, the large majority of works focus on RGB images. Labbe et al [17] proposed a method that, given a single RGB image of a known articulated robot, estimates the 6D camera-to-robot pose in terms of rigid translation and rotation through a render-and-compare approach. A reference point and an anchor part are needed to perform the estimation, and their choice significantly affects the performance.…”

Section: Related Work a Robot Pose Estimation (Rpe)mentioning

confidence: 99%

Semi-Perspective Decoupled Heatmaps for 3D Robot Pose Estimation from Depth Maps

Simoni¹,

Pini²,

Borghi³

et al. 2022

Preprint

View full text Add to dashboard Cite

Knowing the exact 3D location of workers and robots in a collaborative environment enables several real applications, such as the detection of unsafe situations or the study of mutual interactions for statistical and social purposes. In this paper, we propose a non-invasive and light-invariant framework based on depth devices and deep neural networks to estimate the 3D pose of robots from an external camera. The method can be applied to any robot without requiring hardware access to the internal states. We introduce a novel representation of the predicted pose, namely Semi-Perspective Decoupled Heatmaps (SPDH), to accurately compute 3D joint locations in world coordinates adapting efficient deep networks designed for the 2D Human Pose Estimation. The proposed approach, which takes as input a depth representation based on XYZ coordinates, can be trained on synthetic depth data and applied to real-world settings without the need for domain adaptation techniques. To this end, we present the SimBa dataset, based on both synthetic and real depth images, and use it for the experimental evaluation. Results show that the proposed approach, made of a specific depth map representation and the SPDH, overcomes the current state of the art.

show abstract

“…Data augmentation is a huge field (e.g., Shorten and Khoshgoftaar (2019)) with various techniques and in-depth discussion of each of these techniques is out of scope of this paper. Nevertheless, we can say that the techniques such as background augmentation, adding noise or cropping/transforming images, are common means to increasing the data variation in the source domain (Lambrecht and Kästner (2019); Gulde et al (2019); Lee et al (2020); Labbe et al (2021)). The model is then trained under more varied conditions which helps improve generalization and break the dependence on annotated data from the target domain.…”

Section: Domain Augmentationmentioning

confidence: 99%

Visual state estimation in unseen environments through domain adaptation and metric learning

2022

View full text Add to dashboard Cite

In robotics, deep learning models are used in many visual perception applications, including the tracking, detection and pose estimation of robotic manipulators. The state of the art methods however are conditioned on the availability of annotated training data, which may in practice be costly or even impossible to collect. Domain augmentation is one popular method to improve generalization to out-of-domain data by extending the training data set with predefined sources of variation, unrelated to the primary task. While this typically results in better performance on the target domain, it is not always clear that the trained models are capable to accurately separate the signals relevant to solving the task (e.g., appearance of an object of interest) from those associated with differences between the domains (e.g., lighting conditions). In this work we propose to improve the generalization capabilities of models trained with domain augmentation by formulating a secondary structured metric-space learning objective. We concentrate on one particularly challenging domain transfer task—visual state estimation for an articulated underground mining machine—and demonstrate the benefits of imposing structure on the encoding space. Our results indicate that the proposed method has the potential to transfer feature embeddings learned on the source domain, through a suitably designed augmentation procedure, and on to an unseen target domain.

show abstract

Single-view robot pose and joint angle estimation via render & compare

Cited by 36 publications

References 47 publications

AlignSDF: Pose-Aligned Signed Distance Fields for Hand-Object Reconstruction

AlignSDF: Pose-Aligned Signed Distance Fields for Hand-Object Reconstruction

Semi-Perspective Decoupled Heatmaps for 3D Robot Pose Estimation from Depth Maps

Visual state estimation in unseen environments through domain adaptation and metric learning

Contact Info

Product

Resources

About