Depth and thermal sensor fusion to enhance 3D thermographic reconstruction

Abstract

Three-dimensional geometrical models with incorporated surface temperature data provide important information for various applications such as medical imaging, energy auditing, and intelligent robots. In this paper we present a robust method for mobile and real-time 3D thermographic reconstruction through depth and thermal sensor fusion. A multimodal imaging device consisting of a thermal camera and a RGB-D sensor is calibrated geometrically and used for data capturing. Based on the underlying principle that temperature information remains robust against illumination and viewpoint changes, we present a Thermal-guided Iterative Closest Point (T-ICP) methodology to facilitate reliable 3D thermal scanning applications. The pose of sensing device is initially estimated using correspondences found through maximizing the thermal consistency between consecutive infrared images. The coarse pose estimate is further refined by finding the motion parameters that minimize a combined geometric and thermographic loss function. Experimental results demonstrate that complimentary information captured by multimodal sensors can be utilized to improve performance of 3D thermographic reconstruction. Through effective fusion of thermal and depth data, the proposed approach generates more accurate 3D thermal models using significantly less scanning data.

Sparse Depth Enhanced Direct Thermal-Infrared SLAM Beyond the Visible Spectrum

“…Recent studies have used the thermographic error model to improve robustness. More recently, methods have been introduced that combine thermographic error models to improve robustness [21], [22]. However, these methods that heavily rely on RGB-D sensors and are less suitable for an outdoor environment.…”

Encoder-Decoder Structure With the Feature Pyramid for Depth Estimation From a Single Image

“…Estimating the dense and accurate depth of a scene from a single RGB image is one of the fundamental problems of computer vision and essential for various applications, such as scene understanding [1][2][3][4], 3D modeling [5,6], robotics [7,8], virtual reality [9], and autonomous driving [10]. Given the training set RGB image and the corresponding depth map of the image, depth prediction can be regarded as a pixel-level regression problem; that is, the model directly learns to predict the depth corresponding to each pixel in the single image.…”

High-Accuracy Globally Consistent Surface Reconstruction Using Fringe Projection Profilometry

“…Differently, Cao et al [17] and Yue et al [18] optimized the accumulated error by first identifying the loop closures formed through successful 3D registration between each current frame and other earlier frames, and then performing a pose graph optimization [19] to reduce the sensor poses drifts. However, in their works the loop closures are identifying either by manually checking the 3D point cloud overlapping ratio [17], or by using the measurement system setup information [18], which prevents their further usage in a practical 3D scanning system. Moreover, the pose graph optimization in [17,18] only optimized the inconsistency between two associated sensor poses and their relative pose constraint; it ignores important surface consistency information in the 3D registration process [6].…”

“…However, in their works the loop closures are identifying either by manually checking the 3D point cloud overlapping ratio [17], or by using the measurement system setup information [18], which prevents their further usage in a practical 3D scanning system. Moreover, the pose graph optimization in [17,18] only optimized the inconsistency between two associated sensor poses and their relative pose constraint; it ignores important surface consistency information in the 3D registration process [6].…”