Towards people detection from fused time-of-flight and thermal infrared images

Abstract

ABSTRACT:Obtaining accurate 3d descriptions in the thermal infrared (TIR) is a quite challenging task due to the low geometric resolutions of TIR cameras and the low number of strong features in TIR images. Combining the radiometric information of the thermal infrared with 3d data from another sensor is able to overcome most of the limitations in the 3d geometric accuracy. In case of dynamic scenes with moving objects or a moving sensor system, a combination with RGB cameras of Time-of-Flight (TOF) cameras is suitable. As a TOF camera is an active sensor in the near infrared (NIR) and the thermal infrared camera captures the radiation emitted by the objects in the observed scene, the combination of these two sensors for close range applications is independent from external illumination or textures in the scene. This article is focused on the fusion of data acquired both with a time-of-flight (TOF) camera and a thermal infrared (TIR) camera. As the radiometric behaviour of many objects differs between the near infrared used by the TOF camera and the thermal infrared spectrum, a direct co-registration with feature points in both intensity images leads to a high number of outliers. A fully automatic workflow of the geometric calibration of both cameras and the relative orientation of the camera system with one calibration pattern usable for both spectral bands is presented. Based on the relative orientation, a fusion of the TOF depth image and the TIR image is used for scene segmentation and people detection. An adaptive histogram based depth level segmentation of the 3d point cloud is combined with a thermal intensity based segmentation. The feasibility of the proposed method is demonstrated in an experimental setup with different geometric and radiometric influences that show the benefit of the combination of TOF intensity and depth images and thermal infrared images.

Evaluation of Methods for Coregistration and Fusion of Rpas-Based 3d Point Clouds and Thermal Infrared Images

“…This method implies a mainly planar scene to avoid mismatches; (ii) coregistration of both the dense 3D point clouds from RGB images and from TIR images by coregistering 2D image projections of both point clouds (Hoegner et al, 2014;Urban and Weinmann, 2015); (iii) coregistration based on 2D line segments in every single TIR image and 3D line segments extracted from intersections of planes fitted in the segmented dense 3D point cloud ; (iv) coregistration of both the dense 3D point clouds from RGB images and from TIR images using both ICP (Besl and McKay, 1992;Rusinkiewicz and Levoy, 2001;Chetverikov et al, 2005) and adapted version based on corresponding segmented planes (Hebel and Stilla, 2009); (v) coregistration of both image blocks based on SIFT features (Lowe, 2004). The quality is measured by comparing the differences of the back projection of homologous points in both corrected RGB and TIR images.…”

“…As described in (Borrmann et al, 2013) and (Hoegner et al, 2014), a combination of 3D point clouds and thermal infrared images can be used for building inspections as well. Both methods show limitations in their applicability.…”

Towards the Influence of a Car Windshield on Depth Calculation With a Stereo Camera System

“…Point-to-point correspondences are searched in rectified image pairs to calculate the disparity and to derive the depth. Without using stereo cameras, depth estimation is also possible with a single depth-camera measuring the time-of-flight between camera and objects (Hoegner et al, 2014). The estimated depth is normally represented by a depth map (Zhu et al, 2008).…”

Automatic 3D reconstruction and texture extraction for 3D building models from thermal infrared image sequences.

“…As described in [20], a combination of 3D point clouds and thermal infrared images can be used for building inspections as well. Both methods show limitations in their applicability.…”

“…Both 3d reconstruction and texturing are influenced by various conditions as the thermal radiation of facades depends on temperature differences between inside and outside, weather conditions, and materials. To overcome limitations in the 3d accuracy of thermal infrared based 3d points, a combination of thermal infrared cameras and 3d recording systems like time-of-flight cameras [20] is possible.…”