Depth Quality-Inspired Feature Manipulation for Efficient RGB-D Salient Object Detection

Abstract

RGB-D salient object detection (SOD) recently has attracted increasing research interest by benefiting conventional RGB SOD with extra depth information. However, existing RGB-D SOD models often fail to perform well in terms of both efficiency and accuracy, which hinders their potential applications on mobile devices and real-world problems. An underlying challenge is that the model accuracy usually degrades when the model is simplified to have few parameters. To tackle this dilemma and also inspired by the fact that depth quality is a key factor influencing the accuracy, we propose a novel depth quality-inspired feature manipulation (DQFM) process, which is efficient itself and can serve as a gating mechanism for filtering depth features to greatly boost the accuracy. DQFM resorts to the alignment of low-level RGB and depth features, as well as holistic attention of the depth stream to explicitly control and enhance cross-modal fusion. We embed DQFM to obtain an efficient light-weight model called DFM-Net, where we also design a tailored depth backbone and a two-stage decoder for further efficiency consideration. Extensive experimental results demonstrate that our DFM-Net achieves state-of-the-art accuracy when comparing to existing non-efficient models, and meanwhile runs at 140ms on CPU (2.2× faster than the prior fastest efficient model) with only ∼8.5Mb model size (14.9% of the prior lightest). Our code will be available at https://github.com/zwbx/DFM-Net.

Depth-Quality Purification Feature Processing for Red Green Blue-Depth Salient Object Detection

“…A total of 1700 samples from NJU2K and 800 samples from NLPR were used for training, and tests were performed on STERE, SIP, NLPR, LFSD, NJU2K, and RGBD135. The results of DQPFPNet were compared to those of 16 state-of-the-art (SOTA) models, including C2DF [56], S2MA [33], JL-DCF [30], CoNet [57], UCNet [22], CIRNet [58], SSLSOD [59], cmMS [60], DANet [36], DCF [61], ATSA [62], DSA2F [63], PGAR [64], A2dele [37], MSal [65], and DFMNet [66], as shown in Table 1. The salient maps for the other models were derived from their released predictions, if available, or produced from their public code.…”

Self-Supervised Pretraining for RGB-D Salient Object Detection

“…Generally speaking, the depth map can be utilized in three ways: early fusion (Peng et al 2014;Song et al 2017;Zhao et al 2020b), middle fusion (Feng et al 2016) and late fusion (Fan, Liu, and Sun 2014). According to the number of encoding streams, RGB-D SOD methods can divided into two-stream (Zhao et al 2019;Chen and Fu 2020;Fan et al 2020b;Liu, Zhang, and Han 2020;Zhang et al 2020aZhang et al , 2021bSun et al 2021;Ji et al 2021a) and singlestream (Zhao et al 2020b)…”

“…For fair comparisons, the proposed algorithm is compared with 16 state-of-the-art methods that have released codes, including S2MA (Liu, Zhang, and Han 2020), DANet (Zhao et al 2020b), CMW (Li et al 2020), BBS (Fan et al 2020b), CoNet (Ji et al 2020), UCNet (Zhang et al 2020a), PGAR (Chen and Fu 2020), HDF (Pang et al 2020a), CDNet (Jin et al 2021), DFM (Zhang et al 2021b), DSA2F (Sun et al 2021), DCF (Ji et al 2021a) and HAI (Li et al 2021a). Saliency maps of these competitors are directly provided by their respective authors or computed by their released codes.…”

“…With the development of deep convolutional neural networks (CNNs), many CNNs-Based SOD methods (Zhang et al 2019(Zhang et al , 2020bLi et al 2021b;Liu, Zhang, and Han 2020;Fan et al 2020b;Li et al 2020;Zhang et al 2020aZhang et al , 2021bSun et al 2021;Li et al 2021a;Zhao et al 2020a;Pang et al 2020b;Ji et al 2021b;Li et al 2022;Zhang et al 2021a;Pang et al 2020a) can achieve satisfactory performance. They all are required to be pretrained on the ImageNet (Deng et al 2009) to learn rich and high-performance visual representations for downstream tasks.…”