Learning-Based, Automatic 2D-to-3D Image and Video Conversion

Abstract

Despite a significant growth in the last few years, the availability of 3D content is still dwarfed by that of its 2D counterpart. To close this gap, many 2D-to-3D image and video conversion methods have been proposed. Methods involving human operators have been most successful but also time-consuming and costly. Automatic methods, which typically make use of a deterministic 3D scene model, have not yet achieved the same level of quality for they rely on assumptions that are often violated in practice. In this paper, we propose a new class of methods that are based on the radically different approach of learning the 2D-to-3D conversion from examples. We develop two types of methods. The first is based on learning a point mapping from local image/video attributes, such as color, spatial position, and, in the case of video, motion at each pixel, to scene-depth at that pixel using a regression type idea. The second method is based on globally estimating the entire depth map of a query image directly from a repository of 3D images ( image+depth pairs or stereopairs) using a nearest-neighbor regression type idea. We demonstrate both the efficacy and the computational efficiency of our methods on numerous 2D images and discuss their drawbacks and benefits. Although far from perfect, our results demonstrate that repositories of 3D content can be used for effective 2D-to-3D image conversion. An extension to video is immediate by enforcing temporal continuity of computed depth maps.

Single Image Depth Estimation With Integration of Parametric Learning and Non-Parametric Sampling

“…Also, in future works, we will extend the proposed method by jointly training parametric and non-parametric methods. [18], (d) Make3D [12], (e) Eigen et al [13], (f) ours.…”

“…Batch normalization solves the problem by normalizing layer's input. We compare our method with one non-parametric method [18] and two parametric methods [12,13]. For the quantitative evaluation, we measure several kinds of errors, which are standard measures in depth estimation.…”

“…Contrary to Depth Transfer and Depth Analogy, Konrad et al [18] proposes depth estimation method without warping process. They assume that the location of some objects (i.e., sky, building, furniture) are quite consistent with the candidate images.…”

Learning-based depth estimation from 2D images using GIST and saliency

“…We compute the average and median value of metric C obtained in the LOOCV test across all images in the Kinect- NYU dataset and compare the proposed approach with the Depth Transfer approach by Karsch et al [6], the HOG-based Depth Learning solution of Konrad et al [8] and the LBPBased Learning algorithm of Herrera et al [9]. The results are shown in Table 1, where, as can be observed, the proposed approach outperforms the results of the other state-of-the-art methods for both the average the median of the metric C. This improvement of the results is attributed to the use of the GIST features, and the saliency based weights used to select the k most similar images.…”

Edge-based depth gradient refinement for 2D to 3D learned prior conversion

“…In [4], Karsch et al used an approach based on Scale Invariant Feature Transform (SIFT) flow and an optimization post-process to improve the results and extend the method to work with videos. Konrad et al [5] proposed a more computationally efficient method using a descriptor based on Histogram of Oriented Gradients (HOG) to match similar images instead of the SIFT flow approach. Also a Joint Bilateral Filter is used to enhance the resulting depth map.…”