se(3)-TrackNet: Data-driven 6D Pose Tracking by Calibrating Image Residuals in Synthetic Domains

Abstract

Tracking the 6D pose of objects in video sequences is important for robot manipulation. This work presents se(3)-TrackNet, a data-driven optimization approach for longterm, 6D pose tracking. It aims to identify the optimal relative pose given the current RGB-D observation and a synthetic image conditioned on the previous best estimate and the object's model. The key contribution in this context is a novel neural network architecture, which appropriately disentangles the feature encoding to help reduce domain shift, and an effective 3D orientation representation via Lie Algebra. Consequently, even when the network is trained solely with synthetic data can work effectively over real images. Comprehensive experiments over multiple benchmarks show se(3)-TrackNet achieves consistently robust estimates and outperforms alternatives, even though they have been trained with real images. The approach runs in real time at 90.9Hz. Code, data and supplementary video for this project are available at 1 https://github.com/wenbowen123/iros20-6d-pose-tracking 1 This work has been previously accepted to appear at IEEE IROS 2020 [27]. This is a CVPRW'21 paper on 3D Vision and Robotics.

Vision-driven Compliant Manipulation for Reliable; High-Precision Assembly Tasks

“…This work leverages recent advances in visual tracking that employ temporal cues to dynamically update the 6D pose of tracked objects. In particular, recent work in visual tracking [15] achieves robust and accurate enough estimates at a low latency to work with a wide range of objects. This allows easy integration of visual tracking with planning and control for closing the feedback loop.…”

“…In order to generate the paired input image I t−1 for the network, the process randomly samples a Gaussian relative motion transformation T t t−1 ∈ SE(3) centered on the identity relative transform to render the prior frame I t−1 . During training, data augmentations involving random HSV shift, Gaussian noise, Gaussian blur, and depth-sensing corruption are applied to the RGB and depth data in frame I t , following bi-directional domain alignment techniques [15]. Training on synthetic data takes 250 epochs and is readily applicable to real world scenarios without fine-tuning.…”

“…Estimating the task's state dynamically, such as the object's 6D pose [13,14,15], when onboard sensing is unavailable can be achieved through feedback from inexpensive, external alternatives, such as RGBD cameras [16,17]. Although requiring additional computation, this sensing modality is advantageous as it does not require invasive, bulky sensor suites on the robot, while also providing a wider "field of sensing" for perceiving an extended workspace.…”

“…• Vision-based Object Pose Tracker: Utilizes state-of-theart low-latency RGBD-based 6D object pose tracking [15], which is shown to be accurate enough when combined with mechanical compliance to solve the target tasks. The visual tracking monitors the task's state in the form of the object's 6D pose in a way that is robust to occlusions and varied lighting conditions, while trained solely offline on simulated data (Sec.…”

CSA6D: Channel-Spatial Attention Networks for 6D Object Pose Estimation

“…PoseCNN [10] proposed a network that regresses the center of object and regresses the 3D center distance from the camera directly. TrackNet [35] and DeepIM [36] captured the discrepancy between the current and previous images and used a network to estimate the pose residuals. But they required an initial pose to be estimated at the start of the process in order to make iterative refinement.…”