An AR Geo-Registration Algorithm for UAV TIR Video Streams Based on Dual-Antenna RTK-GPS

Abstract

In emergency response and disaster rescue, unmanned aerial vehicles (UAVs) onboard thermal infrared (TIR) sensors are an essential means of acquiring ground information in the nighttime working environment. To enable field personnel to make better decisions based on TIR video streams returned from a UAV, the geographic information enhancement of TIR video streams is required. At present, it is difficult for low-cost UAVs to carry high-precision attitude sensors and thus obtain high-precision camera attitude information to meet the enhanced processing requirements of UAV TIR video streams. To this end, this paper proposes an improved Kalman filter algorithm to improve the geographic registration (geo-registration) accuracy by fusing the positioning and heading data from the dual-antenna real-time kinematic global positioning system (RTK-GPS) with onboard internal measurement unit (IMU) data. This method can yield high-precision position and attitude data in real time based on low-cost UAV hardware, based on which high-precision geo-registration results can be obtained. The computational complexity can be reduced compared with video stream feature tracking algorithms. Furthermore, the problem of unstable matching due to the low resolution and texture level of TIR video streams can be avoided. The experimental results prove that the proposed method can reduce the registration error by 66.15%, and significantly improve the geo-registration accuracy of UAV TIR video streams. Thus, it can strongly support the popularization and practicality of the application of augmented reality (AR) technology to low-cost UAV platforms.

Complementarity, Interoperability, and Level of Integration of Humanitarian Drones with Emerging Digital Technologies: A State-of-the-Art Systematic Literature Review of Mathematical Models

“…A study [192] showcases the combined utilization of 3D VR, WSNs, cloud computing, and RCPSs for natural disaster management, where the datasets generated by the WSN and the observations of the robotic systems were used to create a realistic VR environment able to reflect actual conditions and facilitate the preparation of rescue scenarios within an acceptable time window. Last, in [224], an algorithm to improve the geographic registration accuracy of UAV imagery was designed, in which AR is implemented in thermal infrared video streams, obtained for the acquisition of information at night by collecting thermal radiation from ground objects without additional lighting measures.…”

“…In area coverage operations, UAVs are equipped with optical [199,201,202,204,209,215,216,222] or thermal cameras [141,224] for them to be able to perform the remote sensing of complex disaster environments. The models in [202,222,224] considered the azimuth or elevation [141,199,202] angles between the UAV and the camera.…”

“…In area coverage operations, UAVs are equipped with optical [199,201,202,204,209,215,216,222] or thermal cameras [141,224] for them to be able to perform the remote sensing of complex disaster environments. The models in [202,222,224] considered the azimuth or elevation [141,199,202] angles between the UAV and the camera. Several studies focused on the processing of the UAV images acquired from such operations, for map generation [215], image classification to eliminate redundant photographs [124,209], damage assessment [199], image segmentation for the detection of flooded areas [204], intelligent recognition of damaged poles [202], as well as image de-noising and feature extraction for accurate disaster identification [201].…”

“…The first one is based on a height-adaptive PID controller using the current position of the landing platform as target, while the second one combines this controller with a Kalman filter to predict the future positions of the platform and provide them as the input to the controller. According to the authors in [224], in the traditional Kalman filter algorithm, most input sensor data are unreliable. This realization led them to develop an improved extended Kalman filter algorithm with real-time kinematic global positioning system (RTK-GPS) data, which are usually highly accurate and can be approximated as accurate measurements relative to the usual drone operating range.…”