Employing Bilinear Fusion and Saliency Prior Information for RGB-D Salient Object Detection

Abstract

Multi-modal feature fusion and saliency reasoning are two core sub-tasks of RGB-D salient object detection. However, most existing models employ linear fusion strategies (e.g., concatenation) for multi-modal feature fusion and use a simple coarse-to-fine structure for saliency reasoning. Despite their simpleness, they can neither fully capture the cross-modal complementary information nor exploit the multi-level complementary information among the cross-modal features at different levels. To address these issues, a novel RGB-D salient object detection model is presented, where we pay special attention to the aforementioned two sub-tasks. Concretely, a multi-modal feature interaction module is first presented to explore more interactions between the unimodal RGB and depth features. It helps to capture their cross-modal complementary information by jointly using some simple linear fusion strategies and bilinear fusion ones. Then, a saliency prior information guided fusion module is presented to exploit the multi-level complementary information among the fused cross-modal features at different levels. Instead of employing a simple convolutional layer for the final saliency prediction, a saliency refinement and prediction module is designed to better exploit those extracted multilevel cross-modal information for RGB-D saliency detection. Experimental results on several benchmark datasets verify the effectiveness and superiority of the proposed framework over some state-of-the-art methods.Index Terms-RGB-D salient object detection, bilinear fusion strategy, saliency prior information guided fusion, saliency refinement and prediction. [9] and segmentation [10]. Benefiting from the progress of Convolutional Neural Networks (CNNs), CNNs based RGB SOD models [2], [11], [12], [13] have significantly improved the performance of conventional hand-crafted feature based approaches [14], [15], [16], [17].However, such algorithms are found vulnerable to complex environments, varying illuminations or cluttered backgrounds. After paying a lot of efforts, researchers realize that using RGB images only cannot solve those challenges. Meanwhile,

Middle-level Fusion for Lightweight RGB-D Salient Object Detection

“…Furthermore, experimental results on several benchmarks show that, by virtue of the proposed IMFF and LFDF modules, our proposed model can make up the performance drop caused by reducing parameters to some extents. Methods JCUF [39] UCNet [57] DMRA [53] SSF [33] BBSNet [34] BIANet [37] EBFS [32] A2dele [42] DANet [41] CoNet [35] ATSA [40]…”

“…Feature-level fusion based models [20], [22], [32]- [34], [38], [39] first extract the unimodal RGB and depth features from the input RGB and depth images, respectively, and then fuse them to capture their complementary information for SOD. For example, JCUF [39] first employed two subnetwork to extract unimodal RGB and depth features from the input RGB and depth images, respectively, and then designed a multi-branch feature fusion module to jointly use the fused cross-modal features and the unimodal RGB and depth features for SOD.…”

“…The proposed model and some existing state-of-the-art RGB-D salient object detection models are evaluated on four benchmark datasets: NJU2000 [25], NLPR [51], STEREO [52] and SIP [26]. The following state-of-the-art models include PCA [20], TSAA [22], CPFP [23], D3Net [26], JCUF [39], ICNet [29], ASIFN [28], UCNet [57], DMRA [53], SSF [33], JLDCF [36], BBSNet [34], BIANet [37], EBFS [32], A2dele [42], DANet [41] and CoNet [35]. Among that, A2dele [42], DANet [41] and CoNet [35] are lightweight models.…”

SwinNet: Swin Transformer Drives Edge-Aware RGB-D and RGB-T Salient Object Detection

“…For RGB-D SOD, our model is compared with several SOTA RGB-D SOD algorithms, including D3Net [78], ASIF-Net [36], ICNet [89], DCMF [52], DRLF [90], SSF [43], SSMA [38], A2dele [46], UC-Net [91], JL-DCF [92], CoNet [44], DANet [81], EBFSP [93],CDNet [94], HAINet [95], RD3D [49], DSA2F [48], MMNet [63] and VST [6]. To ensure the fairness of the comparison results, the saliency maps of the evaluation are provided by the authors or generated by running source codes.…”

TriTransNet: RGB-D Salient Object Detection with a Triplet Transformer Embedding Network

“…Our model is compared with 16 state-of-the-art RGB-D SOD models, including D3Net [22], ICNet [41], DCMF [6], DRLF [67], SSF [81], SSMA [43], A2dele [57], UCNet [80], CoNet [33], DANet [90], JLDCF [24], EBFSP [31],CDNet [35], HAINet [40], RD3D [10] and DSA2F [61]. To ensure the fairness of the comparison results, the saliency maps of the evaluation are provided by the authors or generated by running source codes.…”