MV6D: Multi-View 6D Pose Estimation on RGB-D Frames Using a Deep Point-wise Voting Network

Abstract

Estimating 6D poses of objects is an essential computer vision task. However, most conventional approaches rely on camera data from a single perspective and therefore suffer from occlusions. We overcome this issue with our novel multi-view 6D pose estimation method called MV6D which accurately predicts the 6D poses of all objects in a cluttered scene based on RGB-D images from multiple perspectives. We base our approach on the PVN3D network that uses a single RGB-D image to predict keypoints of the target objects. We extend this approach by using a combined point cloud from multiple views and fusing the images from each view with a DenseFusion layer. In contrast to current multi-view pose detection networks such as CosyPose, our MV6D can learn the fusion of multiple perspectives in an end-to-end manner and does not require multiple prediction stages or subsequent fine tuning of the prediction. Furthermore, we present three novel photorealistic datasets of cluttered scenes with heavy occlusions. All of them contain RGB-D images from multiple perspectives and the ground truth for instance semantic segmentation and 6D pose estimation. MV6D significantly outperforms the stateof-the-art in multi-view 6D pose estimation even in cases where the camera poses are known inaccurately. Furthermore, we show that our approach is robust towards dynamic camera setups and that its accuracy increases incrementally with an increasing number of perspectives.

Symmetry-Aware 6D Object Pose Estimation via Multitask Learning

“…[16] improves categorylevel object pose and size prediction by modifying pose consistency learning. [17] uses a deep multi-directional fusion network to efficiently fuse RGB-D frames from multiple perspectives.…”

“…Currently, the academic community is increasingly focusing on deep learning based 6D pose estimation methods. However, most of these methods predominantly rely on RGB [3]- [8] or RGB-D [9], [11]- [17] information because they require the rich features provided by RGB information for subsequent predictions. While these methods might be effective in scenarios involving daily object grasping, they struggle with industrial object grasping becauseof the following reasons.…”

Multi-View Metal Parts Pose Estimation Based on a Single Camera

“…Recent studies on multi-view pose estimation have seen multiple approaches employing neural networks to tackle this task. MV6D predicts the 6D pose from multiple viewpoints and constitutes an end-to-end approach [ 26 ], eliminating the need for multiple prediction stages. Nevertheless, it relies on RGB-D data.…”