3D Object Detection via 2D Segmentation-Based Computational Integral Imaging Applied to a Real Video

Abstract

This study aims to achieve accurate three-dimensional (3D) localization of multiple objects in a complicated scene using passive imaging. It is challenging, as it requires accurate localization of the objects in all three dimensions given recorded 2D images. An integral imaging system captures the scene from multiple angles and is able to computationally produce blur-based depth information about the objects in the scene. We propose a method to detect and segment objects in a 3D space using integral-imaging data obtained by a video camera array. Using objects’ two-dimensional regions detected via deep learning, we employ local computational integral imaging in detected objects’ depth tubes to estimate the depth positions of the objects along the viewing axis. This method analyzes object-based blurring characteristics in the 3D environment efficiently. Our camera array produces an array of multiple-view videos of the scene, called elemental videos. Thus, the proposed 3D object detection applied to the video frames allows for 3D tracking of the objects with knowledge of their depth positions along the video. Results show successful 3D object detection with depth localization in a real-life scene based on passive integral imaging. Such outcomes have not been obtained in previous studies using integral imaging; mainly, the proposed method outperforms them in its ability to detect the depth locations of objects that are in close proximity to each other, regardless of the object size. This study may contribute when robust 3D object localization is desired with passive imaging, but it requires a camera or lens array imaging apparatus.

Efficient depth localization of objects in a 3D space using computational integral imaging

“…Recently, we proposed an algorithm for detecting and localizing objects in a 3D scene using computational integralimaging [7]. We used a custom prototype camera array [13] to obtain an array of 21 images or videos (in a 3 x 7 configuration).…”

“…Following a recently developed method [7], in our approach for objects' depth localization we employ computational integral-imaging [8,9] which is a passive imaging technique that can produce information about the depth of objects in the scene by imaging the scene into an array of images slightly shifted from each other. Computational integral imaging calculates reconstructed depth planes (RPs) of the scene from the array of images as follows [9,10]: , , ) ,…”