Light Field Reconstruction Using Residual Networks on Raw Images

Abstract

Although Light-Field (LF) technology attracts attention due to its large number of applications, especially with the introduction of consumer LF cameras and its frequent use, reconstructing densely sampled LF images represents a great challenge to the use and development of LF technology. Our paper proposes a learning-based method to reconstruct densely sampled LF images from a sparse set of input images. We trained our model with raw LF images rather than using multiple images of the same scene. Raw LF can represent the two-dimensional array of images captured in a single image. Therefore, it enables the network to understand and model the relationship between different images of the same scene well and thus restore more texture details and provide better quality. Using raw images has transformed the task from image reconstruction into image-to-image translation. The feature of small-baseline LF was used to define the images to be reconstructed using the nearest input view to initialize input images. Our network was trained end-to-end to minimize the sum of absolute errors between the reconstructed and ground-truth images. Experimental results on three challenging real-world datasets demonstrate the high performance of our proposed method and its outperformance over the state-of-the-art methods.

RCA-LF: Dense Light Field Reconstruction Using Residual Channel Attention Networks

“…Zhang et al [ 37 ] used micro-lens pictures and view image stacks to investigate further LF data. Salem et al [ 38 ] used the raw LF representation to ease the reconstruction process. In addition, they initialized the input image using the nearest view initialization method.…”

“…Yeung et al [ 46 ] and Liu et al [ 36 ] used MATLAB to build their code, which contains many time-consuming reshaping operations. Compared to Salem et al [ 38 ], they used 15 residual blocks (RBs) compared to the 10 RBs in our proposed work. In addition, they process LFs in the raw representation of size 7 H × 7 W compared to H × W in our implementation.…”

“…However, they neglected the angular LF information, slightly affecting the final reconstruction quality. Salem et al [ 38 ] mapped the LF reconstruction problem from the 4D into the 2D domain by transforming the 4D LF into a 2D raw LF image to ease the reconstruction. They provided satisfactory reconstruction quality using a model inspired by the RCAN [ 39 , 40 ].…”

Novel Projection Schemes for Graph-Based Light Field Coding

“…However, at the time of writing of this paper, the 4DPM mode is not yet available in the open source code of JPEG Pleno Reference Software [33], and thus, the 4DTM mode is used for comparison in this paper instead. From a sparse set of decoded images, a residual network model was proposed in [34] to reconstruct densely sampled LF images. Instead of training a model with sparse sampled viewpoints of the same scene, the raw lenslet image is directly used, and thus, the image reconstruction task is transformed into image-to-image translation.…”

“…The remaining selected views are used to estimate geometry information (Right side), including depth/disparity estimation, and obtaining an efficient graph model. Deep Learning has also been introduced into view synthesis-based LF reconstruction.From a sparse set of decoded images, a residual network model was proposed in[34] to reconstruct densely sampled LF images. Instead of training a model with sparse sampled viewpoints of the same scene, the raw lenslet image is directly used, and thus, the image reconstruction task is transformed into image-to-image translation.…”