EPro-PnP: Generalized End-to-End Probabilistic Perspective-n-Points for Monocular Object Pose Estimation

Abstract

Locating 3D objects from a single RGB image via Perspective-n-Point (PnP) is a long-standing problem in computer vision. Driven by end-to-end deep learning, recent studies suggest interpreting PnP as a differentiable layer, allowing for partial learning of 2D-3D point correspondences by backpropagating the gradients of pose loss. Yet, learning the entire correspondences from scratch is highly challenging, particularly for ambiguous pose solutions, where the globally optimal pose is theoretically non-differentiable w.r.t. the points. In this paper, we propose the EPro-PnP, a probabilistic PnP layer for general end-to-end pose estimation, which outputs a distribution of pose with differentiable probability density on the SE(3) manifold. The 2D-3D coordinates and corresponding weights are treated as intermediate variables learned by minimizing the KL divergence between the predicted and target pose distribution. The underlying principle generalizes previous approaches, and resembles the attention mechanism. EPro-PnP can enhance existing correspondence networks, closing the gap between PnP-based method and the task-specific leaders on the LineMOD 6DoF pose estimation benchmark. Furthermore, EPro-PnP helps to explore new possibilities of network design, as we demonstrate a novel deformable correspondence network with the state-of-the-art pose accuracy on the nuScenes 3D object detection benchmark. Our code is available at https://github.com/tjiiv-cprg/EPro-PnP-v2.

Robot Person Following Under Partial Occlusion

“…While in other sequences where only a partially occluded body is observed, the single-joint-based method always fails to locate the target person as expected. MonoLoco [30] and Mono3DBox [31] also perform poorly with a low recall and a high ALE, especially on Sequence II to IV. Mono3DPose [32] and MonoDepth [37] can work on TABLE I.…”

“…MonoLoc [30] first uses a neural network to detect joints of a person in an image, and then utilizes these estimated 2D joint positions to locate the person in 3D by a multi-task neural network. EPro-PnP [31] describes the pose of a person in the form of a 3D bounding box by integrating learnable 2D-3D correspondences. RootNet [32] develops a top-down pose estimation solution that computes the 3D poses of multiple people with respect to the camera coordinate frame.…”

“…• MonoLoco [30] could locate the person directly by a neural network with the 2D pose of the person as input. • Mono3DBox predicts the 3D bounding box of the person based on EPro-PnP [31] of which the bottom center is utilized as the estimated location. • Mono3DPose performs person location by predicting his 3D root location through RootNet [32].…”

DR-WLC: Dimensionality Reduction cognition for object detection and pose estimation by Watching, Learning and Checking

“…But since the PnP problem is not differentiable at some points [55], it becomes difficult to learn all points and weights in an end-to-end manner. Recently, EPro-PnP proposed by Chen et al [56] makes the PnP problem derivable by introducing a probability density distribution, which greatly enhances the adaptability of the monocular object pose estimation model. The deep learning-based 6DoF pose estimation work no longer relies on 3D template matching [57].…”

Open Challenges for Monocular Single-shot 6D Object Pose Estimation

“…Algorithms for such 2D-3D correspondence registration are either classical ones, like the Perspective-n-Points (PnP) algorithm [54], or learned functions [13], [44], [100], [105].…”