RGBT Salient Object Detection: A Large-Scale Dataset and Benchmark

Tu, Zhengzheng; Ma, Yan; Li, Zhun; Li, Chenglong; Xu, Jieming; Liu, Yongtao

doi:10.1109/tmm.2022.3171688

Cited by 90 publications

(57 citation statements)

References 100 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To verify the superiority of the proposed model, we compare it with twenty state-of-the-art (SOTA) SOD methods, which can be further classified into four categories: (1) three RGB SOD methods including CPD [31], BASNet [33] , and EGNet [32]; (2) six RGB-D SOD methods including DMRA [42], MMCI [93], JL-DCF [44], S2MA [39], DPANet [45], and CDINet [49]; (3) three traditional RGB-T SOD methods including SGDL [65], MTMR [63], and M3S-NIR [64]; (4) eight deep learning RGB-T SOD methods including ADF [81], MMNet [67], MIDD [66], APNet [70], ECFFNet [69], CSRNet [28], CGFNet [68], and MIA [41]. All RGB and RGB-D SOD models are retrained on the same RGB-T training dataset as our model for fair comparison.…”

Section: Comparison With State-of-the-art Methodsmentioning

confidence: 99%

“…This is because we regulate the interaction between the RGB image and the thermal image through the learned global illuminance score, supplement the semantic content for each layer of the thermal image branch in the encoding stage, and take full advantage of the valuable information that the thermal modality can provide in the decoding stage. subsubsectionComputational Complexity To compare the complexity of different algorithms, we select four open-source RGB-T SOD models for comparison, including CGFNet [68], APNet [70], ADF [81], MIDD [66]. Table II shows the FLOPs (Floating Point Operations) and maximum F-measure of different algorithms on the VT1000 dataset.…”

Section: Comparison With State-of-the-art Methodsmentioning

confidence: 99%

“…We use three publicly available RGT-T SOD benchmark datasets to demonstrate the effectiveness of the proposed method, including VT821 dataset [63], VT1000 dataset [65], and VT5000 dataset [81]. The VT821 dataset contains 821 image pairs aligned in different scenes, since the RGB-T images of the VT821 dataset are manually registered, so missing areas will appear in thermal images.…”

Section: Experiments a Datasets And Evaluation Metricsmentioning

confidence: 99%

See 2 more Smart Citations

Does Thermal Really Always Matter for RGB-T Salient Object Detection?

Cong

Zhang

et al. 2023

IEEE Trans. Multimedia

View full text Add to dashboard Cite

In recent years, RGB-T salient object detection (SOD) has attracted continuous attention, which makes it possible to identify salient objects in environments such as low light by introducing thermal image. However, most of the existing RGB-T SOD models focus on how to perform cross-modality feature fusion, ignoring whether thermal image is really always matter in SOD task. Starting from the definition and nature of this task, this paper rethinks the connotation of thermal modality, and proposes a network named TNet to solve the RGB-T SOD task. In this paper, we introduce a global illumination estimation module to predict the global illuminance score of the image, so as to regulate the role played by the two modalities. In addition, considering the role of thermal modality, we set up different cross-modality interaction mechanisms in the encoding phase and the decoding phase. On the one hand, we introduce a semantic constraint provider to enrich the semantics of thermal images in the encoding phase, which makes thermal modality more suitable for the SOD task. On the other hand, we introduce a two-stage localization and complementation module in the decoding phase to transfer object localization cue and internal integrity cue in thermal features to the RGB modality. Extensive experiments on three datasets show that the proposed TNet achieves competitive performance compared with 20 state-of-theart methods. The code and results can be found from the link of https://rmcong.github.io/proj TNet.html.

show abstract

Section: Comparison With State-of-the-art Methodsmentioning

confidence: 99%

Section: Comparison With State-of-the-art Methodsmentioning

confidence: 99%

Section: Experiments a Datasets And Evaluation Metricsmentioning

confidence: 99%

See 1 more Smart Citation

Does Thermal Really Always Matter for RGB-T Salient Object Detection?

Cong

Zhang

et al. 2023

IEEE Trans. Multimedia

View full text Add to dashboard Cite

show abstract

“…RGB-D SOD dataset includes NLPR [109] with 1,000 images, NJU2K [110] with 1,985 images, STERE [111] with 1,000 images, DES [112] with 135 images, SIP [59] with 929 images, and DUT [113] with 1,200 images. RGB-T SOD dataset includes VT821 [72], VT1000 [74], and VT5000 [114]. LF SOD dataset includes LFSD [81] with 100 light field data, HFUT-Lytro [115] with 255 samples, DUTLF-FS [84] with 1,462 light field images.…”

Section: Experiments a Datasetsmentioning

confidence: 99%

HRTransNet: HRFormer-Driven Two-Modality Salient Object Detection

Tang

Liu

Tan

et al. 2023

IEEE Trans. Circuits Syst. Video Technol.

View full text Add to dashboard Cite

High-Resolution Transformer (HRFormer) can maintain high-resolution representation and share global receptive fields. It is friendly towards salient object detection (SOD) in which the input and output have the same resolution. However, two critical problems need to be solved for two-modality SOD. One problem is two-modality fusion. The other problem is the HRFormer output's fusion. To address the first problem, a supplementary modality is injected into the primary modality by using global optimization and an attention mechanism to select and purify the modality at the input level. To solve the second problem, a dual-direction short connection fusion module is used to optimize the output features of HRFormer, thereby enhancing the detailed representation of objects at the output level. The proposed model, named HRTransNet, first introduces an auxiliary stream for feature extraction of supplementary modality. Then, features are injected into the primary modality at the beginning of each multi-resolution branch. Next, HRFormer is applied to achieve forwarding propagation. Finally, all the output features with different resolutions are aggregated by intrafeature and inter-feature interactive transformers. Application of the proposed model results in impressive improvement for driving two-modality SOD tasks, e.g., RGB-D, RGB-T, and light field SOD.https://github.com/liuzywen/HRTransNet

show abstract

“…Similar to previous approaches of combining thermal images with RGB images, Zhengzheng et al [ 62 ] propose fusing RGB images with thermal images to detect objects in adverse conditions. A two-stream convolution neural network architecture generates features from RGB and thermal images.…”

Section: Related Workmentioning

confidence: 99%

Exploiting Concepts of Instance Segmentation to Boost Detection in Challenging Environments

Hashmi

Pagani

Liwicki

et al. 2022

Sensors

View full text Add to dashboard Cite

In recent years, due to the advancements in machine learning, object detection has become a mainstream task in the computer vision domain. The first phase of object detection is to find the regions where objects can exist. With the improvements in deep learning, traditional approaches, such as sliding windows and manual feature selection techniques, have been replaced with deep learning techniques. However, object detection algorithms face a problem when performed in low light, challenging weather, and crowded scenes, similar to any other task. Such an environment is termed a challenging environment. This paper exploits pixel-level information to improve detection under challenging situations. To this end, we exploit the recently proposed hybrid task cascade network. This network works collaboratively with detection and segmentation heads at different cascade levels. We evaluate the proposed methods on three complex datasets of ExDark, CURE-TSD, and RESIDE, and achieve a mAP of 0.71, 0.52, and 0.43, respectively. Our experimental results assert the efficacy of the proposed approach.

show abstract

RGBT Salient Object Detection: A Large-Scale Dataset and Benchmark

Cited by 90 publications

References 100 publications

Does Thermal Really Always Matter for RGB-T Salient Object Detection?

Does Thermal Really Always Matter for RGB-T Salient Object Detection?

HRTransNet: HRFormer-Driven Two-Modality Salient Object Detection

Exploiting Concepts of Instance Segmentation to Boost Detection in Challenging Environments

Contact Info

Product

Resources

About