Light Field Intrinsics with a Deep Encoder-Decoder Network

Abstract

We present a fully convolutional autoencoder for light fields, which jointly encodes stacks of horizontal and vertical epipolar plane images through a deep network of residual layers. The complex structure of the light field is thus reduced to a comparatively low-dimensional representation, which can be decoded in a variety of ways. The different pathways of upconvolution we currently support are for disparity estimation and separation of the lightfield into diffuse and specular intrinsic components. The key idea is that we can jointly perform unsupervised training for the autoencoder path of the network, and supervised training for the other decoders. This way, we find features which are both tailored to the respective tasks and generalize well to datasets for which only example light fields are available. We provide an extensive evaluation on synthetic light field data, and show that the network yields good results on previously unseen real world data captured by a Lytro Illum camera and various gantries.

Fast and Accurate 3D Measurement Based on Light-Field Camera and Deep Learning

“…We use data provided by [13] along with the additional data provided by the benchmark [43] as training data. These two datasets both provide light-field images in the form of SAIs and ground truth depth and disparity maps all in the size of 512 × 512 pixels.…”

“…Furthermore, various recently proposed EPI-based neural networks [13,31,32,33,34] have shown promising performance in light-field depth estimation. Heber et al [32] used Convolutional Neural Networks (CNN) to predict EPI line orientations, and then developed an end-to-end deep network architecture to predict depth [33].…”

“…Heber et al [32] used Convolutional Neural Networks (CNN) to predict EPI line orientations, and then developed an end-to-end deep network architecture to predict depth [33]. Alperovich et al [13] present a fully convolutional autoencoder for light-field images, which can be decoded in a variety of ways to acquire disparity map, diffuse, and specular intrinsic components. Feng et al [34] proposed FaceLFnet based on dense block and EPIs from horizontal and vertical SAIs.…”

“…To further improve depth estimation accuracy for light-field images, challenges induced by small viewing angle of lenslet-based light-field camera need to be properly addressed. A series of algorithms have, therefore, been proposed to solve the occlusions [8,11,12], narrow baseline [6], and intrinsic component recovering [13] difficulties. Although computationally expensive [14], these algorithms have been successfully applied in high-texture, non-reflective, and Lambertian surfaces.…”

Snow Removal From Light Field Images

“…Our proposed LF restoration network is designed based on a 3D U-Net structure with an additional ConvLSTM. In the last few years, the U-Net structure has been successfully utilized for depth estimation [8], [24] and reconstruction [25]- [27] for LF images. On the one hand, a 3D CNN [7], [8] can properly handle all channels of the same subaperture view to avoid undesirable interference between channels of different subaperture views.…”