A Framework for Learning Depth From a Flexible Subset of Dense and Sparse Light Field Views

Abstract

Light field is a type of image data that captures the 3D scene information by recording light rays emitted from a scene at various orientations. It offers a more immersive perception than classic 2D images but at the cost of huge data volume. In this paper, we draw inspiration from the visual characteristics of Sub-Aperture Images (SAIs) of light field and design a compact neural network representation for the light field compression task. The network backbone takes randomly initialized noise as input and is supervised on the SAIs of the target light field. It is composed of two types of complementary kernels: descriptive kernels (descriptors) that store scene description information learned during training, and modulatory kernels (modulators) that control the rendering of different SAIs from the queried perspectives. To further enhance compactness of the network meanwhile retain high quality of the decoded light field, we accordingly introduce modulator allocation and kernel tensor decomposition mechanisms, followed by non-uniform quantization and lossless entropy coding techniques, to finally form an efficient compression pipeline. Extensive experiments demonstrate that our method outperforms other state-of-the-art (SOTA) methods by a significant margin in the light field compression task. Moreover, after aligning descriptors, the modulators learned from one light field can be transferred to new light fields for rendering dense views, indicating a potential solution for view synthesis task.

Bypassing Depth Maps Transmission For Immersive Video Coding

“…The authors of [10] utilize deep learning methods to estimate depth maps for light-fields. Using a fine tuned FlowNet 2.0 network, several candidate depth maps are estimated between the target view and other horizontal or vertical views of variable distance.…”

Scene Flow Estimation From Sparse Light Fields Using a Local 4D Affine Model

“…We can observe that our method always yields the most accurate optical flows (see Table I), and disparity maps (see Table II). Our method is only outperformed by the FDE method [28] for the disparity of the central view on Bamboo2 clean & final and Temple1 clean. However, even for these light fields, our method provides better average results than FDE when considering the disparity maps of all the views, which indicates a better consistency between views.…”

“…A deep neural network, called Dispnet, is proposed in [26] based on the optical flow estimation network Flownet2 [27] but computing 1D correlation instead of 2D correlation to be better suited for disparity estimation. The authors in [28] propose a learning based depth estimation framework suitable for both densely and sparsely sampled light fields, that can learn depth maps for every viewpoint from any subset of input views.…”

“…In the test reported here, we have used ground truth disparity maps for this method, thus showing the best results it can give for the estimated scene flow. We also compare the disparity maps that we estimate, as part of our scene flow model, with the ones obtained with the deep learning based disparity estimation method in [28], referred to here as FDE (Flexible Depth Estimation). Besides, our method is compared with the initial scene flow estimated as in IV-A, using PWC-Net [20].…”

Learning Fused Pixel and Feature-Based View Reconstructions for Light Fields

“…Our training data includes 94 synthetic light field scenes [28,29] and 100 real world scenes captured by a Lytro Illum camera [9]. The model is first trained on synthetic light fields, and then further finetuned with real ones.…”