6D Robotic Assembly Based on RGB-only Object Pose Estimation

Abstract

Comprehending natural language instructions is a critical skill for robots to cooperate effectively with humans. In this paper, we aim to learn 6D poses for robotic assembly by natural language instructions. For this purpose, Language-Instructed 6D Pose Regression Network (LanPose) is proposed to jointly predict the 6D poses of the observed object and the corresponding assembly position. Our proposed approach is based on the fusion of geometric and linguistic features, which allows us to finely integrate multi-modality input and map it to the 6D pose in SE(3) space by the cross-attention mechanism and the language-integrated 6D pose mapping module, respectively. To validate the effectiveness of our approach, an integrated robotic system is established to precisely and robustly perceive, grasp, manipulate and assemble blocks by language commands. 98.09 and 93.55 in ADD(-S)-0.1d are derived for the prediction of 6D object pose and 6D assembly pose, respectively. Both quantitative and qualitative results demonstrate the effectiveness of our proposed languageinstructed 6D pose estimation methodology and its potential to enable robots to better understand and execute natural language instructions.

Fine-grained Load Balancing with Traffic-aware Rerouting in DataCenter Networks

“…The robotic grasping can be classified into three categories: model-based, half-model-based, model-free robotic grasping. Model-based robotic grasping estimates the pose of the object and performs grasping using sampling algorithms or predefined grasping poses [21], [22]. Half model-based grasping methods usually extract the regions of interest with similar features then execute grasping [23].…”