Learning to Synthesize a 4D RGBD Light Field from a Single Image

Abstract

We present a machine learning algorithm that takes as input a 2D RGB image and synthesizes a 4D RGBD light field (color and depth of the scene in each ray direction). For training, we introduce the largest public light field dataset, consisting of over 3300 plenoptic camera light fields of scenes containing flowers and plants. Our synthesis pipeline consists of a convolutional neural network (CNN) that estimates scene geometry, a stage that renders a Lambertian light field using that geometry, and a second CNN that predicts occluded rays and non-Lambertian effects. Our algorithm builds on recent view synthesis methods, but is unique in predicting RGBD for each light field ray and improving unsupervised single image depth estimation by enforcing consistency of ray depths that should intersect the same scene point.

A Dual Camera System for High Spatiotemporal Resolution Video Acquisition

“…Thus camera parallax could be an issue that affects the system performance. We show in our studies below by using simulation data from available KITTI [31], Flower [35], LFVideo [37], Stanford light field [1] datasets, and real data captured by our dual camera system with various parallax settings to demonstrate the robustness of our method.…”

“…Comparison Using Simulation Data: We test our AWnet on Flower [35], LFVideo [37] and Stanford light field (Lego Gantry) [1] datasets following the same configuration in [43]. The Flower and LFVideo datasets are light field images captured using Lytro ILLUM camera.…”

“…Each light field image has 376×541 spatial samples and 14×14 angular samples (grid). The same as the methods applied in [35], [43], we extract the central 8×8 grid of angular samples to avoid invalid images. Parallax is offered by setting the reference image I ref at (0, 0), and associated low-resolution correspondence at (i, i), with 0 < i ≤ 7 by shifting position to another different angular sample.…”

“…For example, Flower(1,1) and LFVideo (7,7) in Table 5, represent low-resolutions at (1,1) and (7,7) with respect to the respective references at (0,0). Images in both Flower and LFVideo datasets exhibit small parallax settings [35], [43]. On the other hand, Stanford light field dataset contains the light field images shot using a Canon Digital Rebel XTi with a canon 10-22 mm lens.…”

SteReFo: Efficient Image Refocusing with Stereo Vision

“…Classical image quality metrics, like PSNR and SSIM, do not fully frame the perceptual quality of refocused images [6]. Because there is no consensus on what, quantitatively, makes for a good refocused images (bokeh-wise but also in terms of object boundaries and physical blur accuracy), subjective assessments are often used [19] and some papers exclusively focus on qualitative assessments [54,52,46,51,45]. In order to provide quantitative evaluation of our results, in addition to classical metrics, we propose to utilize a perceptual metric commonly used by the super resolution community [7,6], the NIQE score [34].…”