Imitation Learning through Image Augmentation Using Enhanced Swin Transformer Model in Remote Sensing

Abstract

In unmanned systems, remote sensing is an approach that collects and analyzes data such as visual images, infrared thermal images, and LiDAR sensor data from a distance using a system that operates without human intervention. Recent advancements in deep learning enable the direct mapping of input images in remote sensing to desired outputs, making it possible to learn through imitation learning and for unmanned systems to learn by collecting and analyzing those images. In the case of autonomous cars, raw high-dimensional data are collected using sensors, which are mapped to the values of steering and throttle through a deep learning network to train imitation learning. Therefore, by imitation learning, the unmanned systems observe expert demonstrations and learn expert policies, even in complex environments. However, in imitation learning, collecting and analyzing a large number of images from the game environment incurs time and costs. Training with a limited dataset leads to a lack of understanding of the environment. There are some augmentation approaches that have the limitation of increasing the dataset because of considering only the locations of objects visited and estimated. Therefore, it is required to consider the diverse kinds of the location of objects not visited to solve the limitation. This paper proposes an enhanced model to augment the number of training images comprising a Preprocessor, an enhanced Swin Transformer model, and an Action model. Using the original network structure of the Swin Transformer model for image augmentation in imitation learning is challenging. Therefore, the internal structure of the Swin Transformer model is enhanced, and the Preprocessor and Action model are combined to augment training images. The proposed method was verified through an experimental process by learning from expert demonstrations and augmented images, which reduced the total loss from 1.24068 to 0.41616. Compared to expert demonstrations, the accuracy was approximately 86.4%, and the proposed method achieved 920 points and 1200 points more than the comparison model to verify generalization.

A Dual Convolutional Neural Network with Attention Mechanism for Thermal Infrared Image Enhancement

“…Wang et al [26] introduced a Target Attention Deep Neural Network (TADNN), which achieves discriminative enhancement in an end-to-end manner. To further improve the enhancement of thermal infrared images, resulting in a more outstanding visual effect in the enhanced images, Park et al [16] propose the use of imitation learning based on the Enhanced Swin Transformer Model for thermal infrared image enhancement. Furthermore, Pang et al [17] leverage a detail enhancement network composed of multiple Convolutional Mixed Attention Blocks (MAB), residual learning (RL), and upsampling units to extract deep features from the input and learn meaningful thermal radiation target information.…”

“…In recent years, enhancement algorithms for thermal infrared images have been continuously evolving and improving. Starting from the early algorithms based on histogram equalization [6][7][8][9][10][11][12][13][14][15], advancements have led to the development of image enhancement techniques utilizing deep learning [16][17][18][19][20][21][22][23][24][25][26]. As a result, the enhanced effectiveness of thermal infrared images has demonstrated noticeable improvements.…”