Light Field Saliency Detection With Deep Convolutional Networks

Abstract

CNN-based methods have been proven to work well for saliency detection on RGB images owing to the outstanding feature representation abilities of CNNs. However, their performance will degrade when detecting multiple saliency regions in highly cluttered or similar backgrounds. To address these problems, in this paper we resort to light field imaging, which records the color intensity of each pixel as well as the directions of incoming light rays, and thus can improve performance for saliency detection owing to the usage of both spatial and angular patterns encoded in light field images. However, it is not trivial to use CNN-based methods for saliency detection on light field images because these methods are not specifically designed for processing light field inputs and current light field datasets are not sufficiently large to train CNNs. To overcome these issues, we first present a new Lytro Illum dataset, which contains 640 light fields and their corresponding micro-lens images, central-viewing images as well as ground-truth saliency maps. Comparing to the current light field saliency datasets [1], [2], the new dataset is larger, of higher quality, contains more variations and more types of light field inputs, which is suitable for training deeper networks as well as better benchmarking algorithms. Furthermore, we propose a novel end-to-end CNNbased framework for light field saliency detection as well as its several variants. We systematically study the impact of different variants and compare light field saliency with regular 2D saliency on the performance of the proposed network. We also conduct extensive experimental comparisons, which indicate that our network significantly outperforms state-of-the-art methods on the proposed dataset and has desired generalization abilities on other existing datasets.

Multi-Generator Adversarial Networks For Light Field Saliency Detection

“…In this way, the texture of salient objects can be well preserved and the computational cost can be largely reduced. In terms of metrics, our method is also superior to state-of-the-art methods on the newly proposed Lytro-Illum dataset [13] and competitive on the other two Lytro datasets [12,17].…”

“…Compared to traditional saliency detection methods [11,12,17], DCNs-based methods [13,14,15,16] learn high-level semantic features automatically from light fields. According to the images used, the DCNsbased methods can be categorized into two categories: multiview-based methods [13,16] and focal-stacks-based methods [14,15]. For example, Piao et al [16] generated multiview images from a single view and proposed a multi-view attention model to predict saliency.…”

“…These works focus on exploiting low-level cues, which is difficult to detect salient objects in complex scenes. There have been few attempts [13,14,15,16] address saliency detection benefiting from the hierarchical feature representation and the weight sharing mechanism of deep convolutional networks (DCNs). However,learning from small-scale light field saliency datasets [12,13,17] will lead to high-order inconsistency between ground truths and predicted maps.…”

A Light Field Sparse and Reconstruction Framework for Improving Rendering Quality

“…In [36], [37], Wu et al took advantage of the clear texture structure of an epipolar plane image (EPI) [38] in light field data and modeled the problem of light field reconstruction from a sparse set of views as a CNNbased angular detail restoration on an EPI. Similarly, the results in [39], [40] also used learning methods to study and improve the quality of light field rendering. Vagharshakyan et al [41], [42] proposed using the shearlet transform to study the sampling and reconstitution of light fields.…”

A Novel Incremental Multi-Template Update Strategy for Robust Object Tracking

“…A generative method refers to describing an object to be tracked in a video through an object-representation method in computer vision and then extracting a corresponding object feature from a current frame containing the object to establish an object template [8][9]. The method then searches the subsequent frames for the area most similar to the object template and gradually iterates to finally achieve the positioning and tracking of the object in the subsequent frames [10][11][12][13][14][15][16][17]. A discriminant method refers to applying both the object template and background information to the tracking system.…”