Semantic Segmentation of UAV Images Based on Transformer Framework with Context Information

Abstract

With the advances in Unmanned Aerial Vehicles (UAVs) technology, aerial images with huge variations in the appearance of objects and complex backgrounds have opened a new direction of work for researchers. The task of semantic segmentation becomes more challenging when capturing inherent features in the global and local context for UAV images. In this paper, we proposed a transformer-based encoder-decoder architecture to address this issue for the precise segmentation of UAV images. The inherent feature representation of the UAV images is exploited in the encoder network using a self-attention-based transformer framework to capture long-range global contextual information. A Token Spatial Information Fusion (TSIF) module is proposed to take advantage of a convolution mechanism that can capture local details. It fuses the local contextual details about the neighboring pixels with the encoder network and makes semantically rich feature representations. We proposed a decoder network that processes the output of the encoder network for the final semantic level prediction of each pixel. We demonstrate the effectiveness of this architecture on UAVid and Urban Drone datasets, where we achieved mIoU of 61.93% and 73.65%, respectively.

Метод Пошуку Об’єктів Інтересу За Спектральними Ознаками На Зображеннях З Активної Оптико-Електронної Системи Спостереження

“…Також слід вказати роботи, в яких обговорюються методи виявлення спектральних аномалій на зображеннях [6; 7] та завдання сегментування зображень [8]. Під спектральною аномалією слід розуміти область невеликого розміру на зображенні, спектральні відмінності пікселів якої суттєво відрізняються від оточуючої її околиці.…”

Improving a method for segmenting optical-electronic images acquired from space observation systems based on the firefly algorithm

“…To solve the problem of semantic segmentation, an improved architecture of the encoder-decoder is proposed in [10]. The advantage of [10] is to solve the problem of precise segmentation of optoelectronic images by combining local contextual details about neighboring pixels in the image with an encoder network.…”

“…To solve the problem of semantic segmentation, an improved architecture of the encoder-decoder is proposed in [10]. The advantage of [10] is to solve the problem of precise segmentation of optoelectronic images by combining local contextual details about neighboring pixels in the image with an encoder network. The main disadvantage is the possibility of capturing inherent characteristics in a local and global context only for images acquired from unmanned aerial vehicles.…”

Semantic Segmentation of Aerial Imagery: A Novel Approach Leveraging Hierarchical Multi-scale Features and Channel-based Attention for Drone Applications

“…This enables drones to make informed decisions based on their environmental understanding ( 6), (7). However, drone semantic segmentation encounters several challenges that musttion be addressed to achieve accurate results (8). The first challenge lies in the variability of scale and perspective inherent in aerial imagery.…”