Multi-View Fusion-Based 3D Object Detection for Robot Indoor Scene Perception

Abstract

To autonomously move and operate objects in cluttered indoor environments, a service robot requires the ability of 3D scene perception. Though 3D object detection can provide an object-level environmental description to fill this gap, a robot always encounters incomplete object observation, recurring detections of the same object, error in detection, or intersection between objects when conducting detection continuously in a cluttered room. To solve these problems, we propose a two-stage 3D object detection algorithm which is to fuse multiple views of 3D object point clouds in the first stage and to eliminate unreasonable and intersection detections in the second stage. For each view, the robot performs a 2D object semantic segmentation and obtains 3D object point clouds. Then, an unsupervised segmentation method called Locally Convex Connected Patches (LCCP) is utilized to segment the object accurately from the background. Subsequently, the Manhattan Frame estimation is implemented to calculate the main orientation of the object and subsequently, the 3D object bounding box can be obtained. To deal with the detected objects in multiple views, we construct an object database and propose an object fusion criterion to maintain it automatically. Thus, the same object observed in multi-view is fused together and a more accurate bounding box can be calculated. Finally, we propose an object filtering approach based on prior knowledge to remove incorrect and intersecting objects in the object dataset. Experiments are carried out on both SceneNN dataset and a real indoor environment to verify the stability and accuracy of 3D semantic segmentation and bounding box detection of the object with multi-view fusion.

Automatic object annotation in streamed and remotely explored large 3D reconstructions

“…Our asynchronous processing of dense 3D reconstruction and object annotation supports the usage of CNNs with long and varying inference time. Thereby, we improve the state of the art, as related work either requires hard-coding of the number of frames that can be interpreted [13,16,20], or only achieves interactive frame rates when using networks with short inference time [17]. Furthermore, our presented approach avoids degradation of the 3D reconstruction due to the asynchronous computation of 3D reconstruction and object detection, both with respect to time and resources.…”

“…With our novel filter pipeline, our framework achieves fast and efficient object classification by highly efficient, generic modification and filtering of the predicted 2D bounding boxes. This makes our approach computationally more lightweight than related works, which use fully convolutional networks [13,15,20] or geometry-based segmentation algorithms [3,15,17] to optimize the network results. Furthermore, our filter pipeline makes our approach independent of complex trained networks as it applies temporal and spatial filters to the standard detection output of a state-of-the-art network.…”

“…While our approach is not as accurate as the geometric object segmentation of Refs. [17,20] it still provides accurate 3D bounding boxes in the virtual 3D reconstruction. Further, we observed a significant increase in accuracy as more frames are interpreted by the CNN.…”

Visual Saliency Detection for Over-Temperature Regions in 3D Space via Dual-Source Images

“…x = bias 1 z = (z 1 + z 2 )/2 light 1 = bias 2 /cosα light 2 = light 1 − bias 1 y = depth depth = light 2 /tanα (12) In the above formula, z is the height position of the heat source, and z 1 and z 2 are the deviations from the origin of the space coordinates at the heights taken at the two positions. In order to reduce the operation error, the average of the two positions is taken as the height deviation.…”

“…Traditionally, using a visible-light binocular camera to reconstruct the target is not possible, because it cannot accurately operate on the abnormal temperature point area [ 7 , 8 , 9 , 10 ]. At present, the most commonly used temperature detection methods use sensor contact measurements [ 11 , 12 , 13 ]. However, there are installation and use problems in engineering applications, so non-contact space measurements can be used to solve the installation problem.…”