Pix2Vox: Context-Aware 3D Reconstruction From Single and Multi-View Images

Abstract

Recovering the 3D representation of an object from single-view or multi-view RGB images by deep neural networks has attracted increasing attention in the past few years. Several mainstream works (e.g., 3D-R2N2) use recurrent neural networks (RNNs) to fuse multiple feature maps extracted from input images sequentially. However, when given the same set of input images with different orders, RNN-based approaches are unable to produce consistent reconstruction results. Moreover, due to long-term memory loss, RNNs cannot fully exploit input images to refine reconstruction results. To solve these problems, we propose a novel framework for single-view and multi-view 3D reconstruction, named Pix2Vox. By using a well-designed encoder-decoder, it generates a coarse 3D volume from each input image. Then, a context-aware fusion module is introduced to adaptively select high-quality reconstructions for each part (e.g., table legs) from different coarse 3D volumes to obtain a fused 3D volume. Finally, a refiner further refines the fused 3D volume to generate the final output. Experimental results on the ShapeNet and Pix3D benchmarks indicate that the proposed Pix2Vox outperforms state-ofthe-arts by a large margin. Furthermore, the proposed method is 24 times faster than 3D-R2N2 in terms of backward inference time. The experiments on ShapeNet unseen 3D categories have shown the superior generalization abilities of our method.

3DMGNet: 3D Model Generation Network Based on Multi-Modal Data Constraints and Multi-Level Feature Fusion

“…In order to verify the effectiveness and advantages of the proposed 3DMGNet, we compare 3DMGNet with state-of-the-art methods, i.e., 3D-R2-N2 [ 23 ], OGN [ 36 ], DRC [ 37 ], Pix2Vox-F [ 38 ], Pix2Vox++/F [ 39 ]. In order to unify the comparison conditions, we follow the same experiment settings as in PixelVox [ 38 ], and compare the IOU results when the threshold is 0.3. The comparison results are shown in Table 4 .…”

“…Compared with 3D-R2-N2 [ 23 ], OGN [ 36 ] and DRC [ 37 ], 3DMGNet can achieve the best generation results, which obtains the highest IOU value, except for the category of Watercraft. The 3DMGNet can achieve better generation accuracy than PixVox-F [ 38 ] and PixVox++/F [ 39 ] in most categories, i.e., Airplane, Bench, Chair, Display, Lamp, Rifle, Sofa and Telephone. Pix2Vox-F and PixVox++/F solves the single-view-based 3D model generation problem by spatial mapping, while the 3DMGNet solves this problem from the perspective of multi-modal feature fusion.…”

“…The 3DMGNet can achieve better generation accuracy than PixVox-F [ 38 ] and PixVox++/F [ 39 ] in most categories, i.e., Airplane, Bench, Chair, Display, Lamp, Rifle, Sofa and Telephone. Pix2Vox-F and PixVox++/F solves the single-view-based 3D model generation problem by spatial mapping, while the 3DMGNet solves this problem from the perspective of multi-modal feature fusion.…”

Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era

“…This can be done in a straightforward manner if the camera parameters are known. Otherwise, the fusion can be done using point cloud registration techniques [80], [81] or fusion networks [82]. Also, point set representations require fixing in advance the number of points N while in methods that use grid representations, the number of points can vary based on the nature of the object but it is always bounded by the grid resolution.…”

“…Also, when given the same set of images with different orders, RNNs are unable to estimate the 3D shape of an object consistently due to permutation variance. To overcome these limitations, Xie et al [82] introduced Pix2Vox, which is composed of multiple encoder-decoder blocks, running in parallel, each one predicts a coarse volumetric grid from its input frame. This eliminates the effect of the order of input images and accelerates the computation.…”