Learning-based view synthesis for light field cameras

Abstract

With the introduction of consumer light field cameras, light field imaging has recently become widespread. However, there is an inherent trade-off between the angular and spatial resolution, and thus, these cameras often sparsely sample in either spatial or angular domain. In this paper, we use machine learning to mitigate this trade-off. Specifically, we propose a novel learning-based approach to synthesize new views from a sparse set of input views. We build upon existing view synthesis techniques and break down the process into disparity and color estimation components. We use two sequential convolutional neural networks to model these two components and train both networks simultaneously by minimizing the error between the synthesized and ground truth images. We show the performance of our approach using only four corner sub-aperture views from the light fields captured by the Lytro Illum camera. Experimental results show that our approach synthesizes high-quality images that are superior to the state-of-the-art techniques on a variety of challenging real-world scenes. We believe our method could potentially decrease the required angular resolution of consumer light field cameras, which allows their spatial resolution to increase.

Super Resolution of Light Field Images Using Linear Subspace Projection of Patch-Volumes

“…[8] and the DCNN based scheme [9]. The angular super-resolution method was also compared against the recent view synthesis method published in [11]. In all the experiments we used the code and pre-trained models provided by the authors or are publicly available on-line.…”

“…Average PSNR gains of 3.7dB, 4.2dB and 2.1dB on the HCI, Stanford and INRIA datasets respectively were obtained for spatial super-resolution using BM+PCA+RR compared to the recent DCNN scheme [9] which obtained the second best results in our experiments. Moreover, subjective results clearly show that the proposed angular super-resolution method provides sharper synthesized intermediate sub-aperture images when compared to those obtained using the DCNN based schemes [9], [11].…”

“…Angular super-resolution methods try to approximate novel sub-aperture images to increase the angular resolution of the light field. We compare our proposed angular super-resolution method with bilinear interpolation between neighbouring sub-aperture images, the SRCNN+LF method in [9] and the view synthesis method presented in [11].…”

“…Experimental results with both synthetic light fields (from the HCI dataset) and real light fields (from the Stanford and the INRIA datasets) show that the proposed method outperforms state-of-the-art single-image super-resolution methods [13], [14], [15] applied on each sub-aperture image separately, and significantly outperfoms prior light field super-resolution algorithms for both spatial [8], [9] and angular [9], [11] super-resolution. Average PSNR gains of 3.7dB, 4.2dB and 2.1dB on the HCI, Stanford and INRIA datasets respectively were obtained for spatial super-resolution using BM+PCA+RR compared to the recent DCNN scheme [9] which obtained the second best results in our experiments.…”

“…This method first employs a spatial DCNN similar to [10] followed by an angular DCNN which synthesizes intermediate views. A cascade of two DCNNs is used in [11] which exploits disparity information to synthesize intermediate sub-aperture images. A sparse representation approach that can be extended for lightfield super-resolution was presented in [12].…”