RGBT Salient Object Detection: A Large-scale Dataset and Benchmark

Abstract

Salient object detection in complex scenes and environments is a challenging research topic. Most works focus on RGB-based salient object detection, which limits its performance of real-life applications when confronted with adverse conditions such as dark environments and complex backgrounds. Taking advantage of RGB and thermal infrared images becomes a new research direction for detecting salient object in complex scenes recently, as thermal infrared spectrum imaging provides the complementary information and has been applied to many computer vision tasks. However, current research for RGBT salient object detection is limited by the lack of a large-scale dataset and comprehensive benchmark. This work contributes such a RGBT image dataset named VT5000, including 5000 spatially aligned RGBT image pairs with ground truth annotations. VT5000 has 11 challenges collected in different scenes and environments for exploring the robustness of algorithms. With this dataset, we propose a powerful baseline approach, which extracts multi-level features within each modality and aggregates these features of all modalities with the attention mechanism, for accurate RGBT salient object detection. Extensive experiments show that the proposed baseline approach outperforms the stateof-the-art methods on VT5000 dataset and other two public datasets. In addition, we carry out a comprehensive analysis of different algorithms of RGBT salient object detection on VT5000 dataset, and then make several valuable conclusions and provide some potential research directions for RGBT salient object detection. Our new VT5000 dataset are made publicly available at https://pan.baidu.com/s/1O5TC-5sEya8N2EGm-xJ5mw Password:o57e.

SwinNet: Swin Transformer Drives Edge-Aware RGB-D and RGB-T Salient Object Detection

“…VT1000 [58] contains 1,000 RGB-T image pairs captured with highly aligned RGB and thermal cameras. VT5000 [59] contains 5,000 pairs of high-resolution, high-diversity and low-deviation RGB-T images. For the sake of fair comparison, we use the same training dataset as in [62], [82], [11], which consists of 2,500 image pairs in VT5000.…”

“…In earlier years, RGB-T SOD adopts machine learning methods, for example, SVM [54], ranking models [55], [56], [57] and graph learning [58]. With the development of CNN, Tu et al [59] propose a baseline model which combines CNN with attention mechanism. Zhang et al [60], [61] propose two end-to-end CNN based RGB-T SOD models to achieve multi-scale, multimodality and multi-level fusion.…”

“…For RGB-T SOD, our model is compared with some SOTA RGB-T SOD algorithms, including MTMR [56], M3S-NIR [55], SGDL [58], ADF [59], ECFFNet [11], MIDD [62], MMNet [63], CSRNet [65], CGFNet [64]. To ensure the fairness of the comparison results, the saliency maps of the evaluation are provided by the authors or generated by running source codes.…”

Scribble-based Boundary-aware Network for Weakly Supervised Salient Object Detection in Remote Sensing Images

“…Most SOD methods [37,9,38,39,11,12] mainly focus on solving the integration of multi-level features to generate a salient map with accurate location and internal consistency. Other works are devoted to studying how to use multi-modal information, such as depth cues [40,41,42] or thermal infrared cues [43,44], as auxiliary inputs of the models to alleviate the defects of individual RGB sources, such as low-light environments and similar texture scenes. It is noted that, some methods [13,17,10,33] also introduce the edge detection as a joint or auxiliary task for SOD, which make the models pay more attention to the object structure and thus improve the localization of salient objects.…”

RGB-D Salient Object Detection: A Survey

“…Therefore, thermal images can provide supplementary information to improve SOD performance when salient objects suffer from varying light, reflective light, or shadows. Some RGB-T models [185]- [193] and datasets (VT821 [187], VT1000 [191] and VT5000 [193]) have already been proposed over the past few years. Similar to RGB-D SOD, the key aim of RGB-T SOD is to fuse RGB and thermal infrared images and exploit the correlations between the two modalities.…”