Learning Multi-modal Information for Robust Light Field Depth Estimation

Abstract

Light field data has been demonstrated to facilitate the depth estimation task. Most learning-based methods estimate the depth information from EPI or sub-aperture images, while less methods pay attention to the focal stack. Existing learning-based depth estimation methods from the focal stack lead to suboptimal performance because of the defocus blur. In this paper, we propose a multi-modal learning method for robust light field depth estimation. We first excavate the internal spatial correlation by designing a context reasoning unit which separately extracts comprehensive contextual information from the focal stack and RGB images. Then we integrate the contextual information by exploiting a attention-guide cross-modal fusion module. Extensive experiments demonstrate that our method achieves superior performance than existing representative methods on two light field datasets. Moreover, visual results on a mobile phone dataset show that our method can be widely used in daily life.

Light field salient object detection: A review and benchmark

“…As a result, it can be viewed as an array of images captured by a grid of cameras. Compared to RGB images captured by a regular camera or depth maps acquired by a depth sensor, the light field data acquired by a plenoptic camera records more comprehensive and complete information about natural scenes, covering, for example, depth information [38][39][40][41][42][43][44], focusness cues [5,42], and angular changes [42,45]. Therefore, light field data can benefit SOD in a number of ways.…”