RGB-D and Thermal Sensor Fusion: A Systematic Literature Review

Brenner, Martin; Reyes, Napoleon H.; Susnjak, Teo; Barczak, Andre L. C.

doi:10.1109/access.2023.3301119

Cited by 16 publications

(6 citation statements)

References 119 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Due to the rarity of thermal images in image processing, largely attributed to the high cost of recording devices, thermal imaging (TI) is often combined with the RGB modality to enhance resolution [21] or for segmentation and detection purposes [22,23]. However, synchronizing and superimposing the different streams is costly [24] and may not achieve perfect alignment. This often results in trade-offs between the resolution of the modalities, varying acquisition rates, and calibration errors.…”

Section: Introductionmentioning

confidence: 99%

ApeTI: A Thermal Image Dataset for Face and Nose Segmentation with Apes

Martin,

Kachel,

Wieg

et al. 2024

Signals

View full text Add to dashboard Cite

The ApeTI dataset was built with the aim of retrieving physiological signals such as heart rate, breath rate, and cognitive load from thermal images of great apes. We want to develop computer vision tools that psychologists and animal behavior researchers can use to retrieve physiological signals noninvasively. Our goal is to increase the use of a thermal imaging modality in the community and avoid using more invasive recording methods to answer research questions. The first step to retrieving physiological signals from thermal imaging is their spatial segmentation to then analyze the time series of the regions of interest. For this purpose, we present a thermal imaging dataset based on recordings of chimpanzees with their face and nose annotated using a bounding box and nine landmarks. The face and landmarks’ locations can then be used to extract physiological signals. The dataset was acquired using a thermal camera at the Leipzig Zoo. Juice was provided in the vicinity of the camera to encourage the chimpanzee to approach and have a good view of the face. Several computer vision methods are presented and evaluated on this dataset. We reach mAPs of 0.74 for face detection and 0.98 for landmark estimation using our proposed combination of the Tifa and Tina models inspired by the HRNet models. A proof of concept of the model is presented for physiological signal retrieval but requires further investigation to be evaluated. The dataset and the implementation of the Tina and Tifa models are available to the scientific community for performance comparison or further applications.

show abstract

Section: Introductionmentioning

confidence: 99%

ApeTI: A Thermal Image Dataset for Face and Nose Segmentation with Apes

Martin,

Kachel,

Wieg

et al. 2024

Signals

View full text Add to dashboard Cite

show abstract

“…The discussion on conducting tests involving RGBD and thermal cameras emphasizes the focus on safety and efficient navigation. The combination of these technologies offers a more comprehensive view of the environment, overcoming the limitations of each individual sensor [44]. The mention of the ability of RGBD cameras to calculate distances and their complementarity with LiDAR distances highlights the synergy between different sensory modalities.…”

Section: Discussionmentioning

confidence: 99%

Robots for Forest Maintenance

Gameiro,

Pereira,

Viegas

et al. 2024

Forests

View full text Add to dashboard Cite

Forest fires are becoming increasingly common, and they are devastating, fueled by the effects of global warming, such as a dryer climate, dryer vegetation, and higher temperatures. Vegetation management through selective removal is a preventive measure which creates discontinuities that will facilitate fire containment and reduce its intensity and rate of spread. However, such a method requires vast amounts of biomass fuels to be removed, over large areas, which can only be achieved through mechanized means, such as through using forestry mulching machines. This dangerous job is also highly dependent on skilled workers, making it an ideal case for novel autonomous robotic systems. This article presents the development of a universal perception, control, and navigation system for forestry machines. The selection of hardware (sensors and controllers) and data-integration and -navigation algorithms are central components of this integrated system development. Sensor fusion methods, operating using ROS, allow the distributed interconnection of all sensors and actuators. The results highlight the system’s robustness when applied to the mulching machine, ensuring navigational and operational accuracy in forestry operations. This novel technological solution enhances the efficiency of forest maintenance while reducing the risk exposure to forestry workers.

show abstract

“…In the feature fusion stage, they address face recognition tasks through concatenation. Martin et al [17] introduce possible research directions after multi-modal fusion with RGB-D and thermal infrared sensors. They discuss various techniques for feature extraction in convolutional models.…”

Section: Pose Estimation Based On Rgb-d Imagesmentioning

confidence: 99%

“…Recent studies suggest that enhancing the capability for cross-modal semantic alignment contributes to improved model performance [15]. One commonly used approach involves employing a branching structure to handle multimodal data [16][17][18][19]. This method processes RGB and depth values in parallel, performing pixel-level feature encoding using a customized network.…”

Section: 、 Introductionmentioning

confidence: 99%

A Modal Fusion Network with Dual Attention Mechanism for 6D Pose Estimation

Chen,

Xie,

Sun

et al. 2024

Preprint

View full text Add to dashboard Cite

The 6D pose estimation based on RGB-D data holds significant application value in computer vision and related fields. Currently, deep learning methods commonly employ convolutional networks for feature extraction, which are sensitive to keypoints at close distances but overlook information related to keypoints at longer distances. Moreover, in subsequent stages, there is a failure to effectively fuse spatial features (depth channel features) and color-texture features (RGB channel features). Consequently, this limitation results in compromised accuracy in existing 6D pose networks based on RGB-D data. In addressing this issue, this paper proposes a novel end-to-end network. Specifically, in the depth value extraction branch, a mask vector attention mechanism is used to establish global spatial weights, achieving robust extraction of depth values. In the fusion stage, a symmetrical fusion module is introduced, which utilizes a cross-attention mechanism to integrate spatial features and color-texture features, enhancing the efficiency of target representation and achieving self-correlated fusion between modalities, thereby effectively improving the accuracy of 6D pose estimation. Experimental evaluations were conducted on the LINEMOD and LINEMOD-OCLUSION datasets, and the results indicate that the proposed method achieved ADD(-S) scores of 95.84% and 47.89%, respectively. Compared to state-of-the-art methods, our approach demonstrates superior performance in pose estimation for objects with complex shapes. Moreover, in the presence of occlusion, the accuracy of pose estimation for asymmetric objects has been effectively improved.

show abstract

RGB-D and Thermal Sensor Fusion: A Systematic Literature Review

Cited by 16 publications

References 119 publications

ApeTI: A Thermal Image Dataset for Face and Nose Segmentation with Apes

ApeTI: A Thermal Image Dataset for Face and Nose Segmentation with Apes

Robots for Forest Maintenance

A Modal Fusion Network with Dual Attention Mechanism for 6D Pose Estimation

Contact Info

Product

Resources

About