A self-supervised learning system for object detection using physics simulation and multi-view pose estimation

Abstract

Progress has been achieved recently in object detection given advancements in deep learning. Nevertheless, such tools typically require a large amount of training data and significant manual effort to label objects. This limits their applicability in robotics, where solutions must scale to a large number of objects and variety of conditions. This work proposes an autonomous process for training a Convolutional Neural Network (CNN) for object detection and pose estimation in robotic setups. The focus is on detecting objects placed in cluttered, tight environments, such as a shelf with multiple objects. In particular, given access to 3D object models, several aspects of the environment are physically simulated. The models are placed in physically realistic poses with respect to their environment to generate a labeled synthetic dataset. To further improve object detection, the network self-trains over real images that are labeled using a robust multi-view pose estimation process. The proposed training process is evaluated on several existing datasets and on a dataset collected for this paper with a Motoman robotic arm. Results show that the proposed approach outperforms popular training processes relying on synthetic -but not physically realistic -data and manual annotation. The key contributions are the incorporation of physical reasoning in the synthetic data generation process and the automation of the annotation process over real images.

An Annotation Saved is an Annotation Earned: Using Fully Synthetic Training for Object Detection

“…Other lines of work utilize photo-realistic rendering and realistic scene compositions to overcome the domain gap by synthesizing images that match the real world as close as possible [9,13,25,17,1,8,33,18]. While these methods have shown promising results they face many hard challenges.…”

Gradual Machine Learning for Entity Resolution

“…Semi-supervised learning (Blum and Mitchell, 1998;Joachims, 1999) addresses this problem by making use of a large amount of unlabeled data and a small amount of labeled data. Similarly, as an autonomous supervised learning approach, self-supervised learning (Mitash et al, 2017) usually extracts and uses the naturally available relevant context and embedded meta data as supervisory signals. Active learning (Arasu et al, 2010;Bellare et al, 2012) is another special case of supervised learning in which a learning algorithm is able to interactively query the user (or some other information source) to obtain the desired outputs at new data points.…”

Improving 6D Pose Estimation of Objects in Clutter Via Physics-Aware Monte Carlo Tree Search

“…Eventhough this is promising as data driven approaches could help close the gap between 3d geometric models and noisy observed data, it needs access to a large model aligned 3d training dataset, which may be difficult to collect. Another technique often used is to perform object segmentation using CNNs trained specifically for the setup [11], [2], [19] and perform point cloud registration methods The image describes the process of hypotheses generation for objects present in the scene. The process starts with extracting object segments S 1:3 using Faster-RCNN [6], followed by using a global point cloud registration technique [4] to compute a set of possible model transformations (T 1:3 ) that corresponds to the respective segments.…”

“…In order to address the issue of potentially conflicting candidate object poses, the scene hypotheses are dynamically constructed by introducing a constrained local optimization step over candidate object poses returned by Super4PCS, a fast global model matching method [4]. To limit detection errors that arise in cluttered scene, the proposed method builds on top of a previous contribution [11], which performs clutter-specific autonomous training to get object segments. This paper provides experimental indications that the set of candidate object poses returned by Super4PCS given the clutter-aware training contains object poses that are close enough to the ground truth, however, these might not be the ones that receive the best matching score according to Super4PCS.…”