A Multi-Task Collaborative Network for Light Field Salient Object Detection

Zhang, Qiudan; Wang, Shiqi; Wang, Xu; Sun, Zhenhao; Jiang, Jianmin

doi:10.1109/tcsvt.2020.3013119

Cited by 49 publications

(15 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…All images in the current light field datasets were generated by processing LFP or LFR files using Lytro Desktop software (http://lightfield-forum.com/ lytro/lytro-archive/), or LFToolbox (http://code .behnam.es/python-lfp-reader/, or https:// ww2.mathworks.cn/matlabcentral/fileexchange/ 75250-light-field-toolbox). Since the raw data cannot be readily utilized, the data forms of light fields used by existing SOD models are diverse, including focal stacks plus all-in-focus images [1, 5, 30-32, 45, 50, 52-55, 58], multi-view images plus center-view images [45,53,60], and micro-lens image arrays [51,59]. As mentioned, depth images can also be synthesized from light field data [38][39][40][41], and therefore can form RGB-D data sources for RGB-D SOD models (see Fig.…”

Section: Forms Of Light Field Datamentioning

confidence: 99%

“…A straightforward approach considers what kind of light field data is utilized, as indicated in Table 1. While four models, DLLF [57], MoLF [31], ERNet [32], LFNet [58] resort to focal stacks, DLSD [45] and MTCNet [60] utilize multi-view images, and MAC [59] uses micro-lens images. Different input data forms often lead to different network designs.…”

Section: Deep Learning-based Modelsmentioning

confidence: 99%

“…Thus, in this paper, we conduct the first comprehensive review and benchmark for light field SOD. We review previous studies on light field SOD, including ten traditional models [1,5,30,[50][51][52][53][54][55][56], seven deep learning-based models [31,32,45,[57][58][59][60], one comparative study [48], and one brief review [49]. In addition, we also review existing light field SOD datasets [5,45,53,57,59], and statistically analyze them, covering object size, distance between object and image center, number of focal slices, and number of objects.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Light field salient object detection: A review and benchmark

Jiang

Zhou

et al. 2022

Comp. Visual Media

View full text Add to dashboard Cite

Salient object detection (SOD) is a long-standing research topic in computer vision with increasing interest in the past decade. Since light fields record comprehensive information of natural scenes that benefit SOD in a number of ways, using light field inputs to improve saliency detection over conventional RGB inputs is an emerging trend. This paper provides the first comprehensive review and a benchmark for light field SOD, which has long been lacking in the saliency community. Firstly, we introduce light fields, including theory and data forms, and then review existing studies on light field SOD, covering ten traditional models, seven deep learning-based models, a comparative study, and a brief review. Existing datasets for light field SOD are also summarized. Secondly, we benchmark nine representative light field SOD models together with several cutting-edge RGB-D SOD models on four widely used light field datasets, providing insightful discussions and analyses, including a comparison between light field SOD and RGB-D SOD models. Due to the inconsistency of current datasets, we further generate complete data and supplement focal stacks, depth maps, and multi-view images for them, making them consistent and uniform. Our supplemental data make a universal benchmark possible. Lastly, light field SOD is a specialised problem, because of its diverse data representations and high dependency on acquisition hardware, so it differs greatly from other saliency detection tasks. We provide nine observations on challenges and future directions, and outline several open issues. All the materials including models, datasets, benchmarking results, and supplemented light field datasets are publicly available at https://github.com/kerenfu/LFSOD-Survey.

show abstract

Section: Forms Of Light Field Datamentioning

confidence: 99%

Section: Deep Learning-based Modelsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Light field salient object detection: A review and benchmark

Jiang

Zhou

et al. 2022

Comp. Visual Media

View full text Add to dashboard Cite

show abstract

“…As described in the SOD review [1], SOD has been extended from RGB image [2], [3], [4] to RGB-D image [5], [6], a group of images [7], [8] and video [9], [10]. Recently, SOD in RGB-T image [11], light field image [12], [13], [14], high-resolution image [15], [16], optical remote sensing image [17], [18], [19] and 360 • omnidirectional image [20], [21] have been gradually researched. SOD can benefit many image and video processing tasks, such as image segmentation [22], [23], tracking [24], [25], [26], retrieval [27], compression [28], cropping [29], [30], retargeting [31], quality assessment [32] and activity prediction [33].…”

Section: Introductionmentioning

confidence: 99%

SwinNet: Swin Transformer Drives Edge-Aware RGB-D and RGB-T Salient Object Detection

Liu

Tan

et al. 2022

IEEE Trans. Circuits Syst. Video Technol.

203

View full text Add to dashboard Cite

Convolutional neural networks (CNNs) are good at extracting contexture features within certain receptive fields, while transformers can model the global long-range dependency features. By absorbing the advantage of transformer and the merit of CNN, Swin Transformer shows strong feature representation ability. Based on it, we propose a cross-modality fusion model, SwinNet, for RGB-D and RGB-T salient object detection.It is driven by Swin Transformer to extract the hierarchical features, boosted by attention mechanism to bridge the gap between two modalities, and guided by edge information to sharp the contour of salient object. To be specific, two-stream Swin Transformer encoder first extracts multi-modality features, and then spatial alignment and channel re-calibration module is presented to optimize intra-level cross-modality features. To clarify the fuzzy boundary, edge-guided decoder achieves inter-level cross-modality fusion under the guidance of edge features. The proposed model outperforms the state-of-theart models on RGB-D and RGB-T datasets, showing that it provides more insight into the cross-modality complementarity task.https://github.com/liuzywen/SwinNet

show abstract

“…Early light field saliency detection works have been dominated by fully supervised methods which require large amounts of accurate pixel-level annotations aligned with the all-focus central view for training [31,40,[56][57][58]. This expensive and time-consuming labelling process hinders the applicability of fully supervised methods to large scale problems.…”

Section: Introductionmentioning

confidence: 99%

Learning from Pixel-Level Noisy Label : A New Perspective for Light Field Saliency Detection

Feng¹,

Zhang²,

Yu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Saliency detection with light field images is becoming attractive given the abundant cues available, however, this comes at the expense of large-scale pixel level annotated data which is expensive to generate. In this paper, we propose to learn light field saliency from pixel-level noisy labels obtained from unsupervised hand crafted featuredbased saliency methods. Given this goal, a natural question is: can we efficiently incorporate the relationships among light field cues while identifying clean labels in a unified framework? We address this question by formulating the learning as a joint optimization of intra light field features fusion stream and inter scenes correlation stream to generate the predictions. Specially, we first introduce a pixel forgetting guided fusion module to mutually enhance the light field features and exploit pixel consistency across iterations to identify noisy pixels. Next, we introduce a cross scene noise penalty loss for better reflecting latent structures of training data and enabling the learning to be invariant to noise. Extensive experiments on multiple benchmark datasets demonstrate the superiority of our framework showing that it learns saliency prediction comparable to state-of-the-art fully supervised light field saliency methods. Our code is available at https://github.com/ OLobbCode/NoiseLF.

show abstract

A Multi-Task Collaborative Network for Light Field Salient Object Detection

Cited by 49 publications

References 43 publications

Light field salient object detection: A review and benchmark

Light field salient object detection: A review and benchmark

SwinNet: Swin Transformer Drives Edge-Aware RGB-D and RGB-T Salient Object Detection

Learning from Pixel-Level Noisy Label : A New Perspective for Light Field Saliency Detection

Contact Info

Product

Resources

About