Training Artificial Intelligence Algorithms with Automatically Labelled UAV Data from Physics-Based Simulation Software

Boone, Jonathan D.; Goodin, C.; Dabbiru, Lalitha; Hudson, Christopher R.; Cagle, Lucas; Carruth, Daniel W.

doi:10.3390/app13010131

Cited by 10 publications

(5 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, data labeling is a costly, time-consuming, and error-prone procedure. Therefore, an automated labeling approach for training convolutional neural networks (CNNs) using pre-obtained data has been studied in a wide range of fields such as medical image processing, aerial photogrammetry, hyperspectral imagery processing, and building information modeling (BIM) [26], [27], [28], [29]. StarCraft II replay data is extracted from PySC2, so in-game information such as the presence or absence of units and buildings can be accurately extracted from each channel in the frame.…”

Section: Proposed Methods a Data Acquisition And Automated Labelingmentioning

confidence: 99%

DESEM: Depthwise Separable Convolution-Based Multimodal Deep Learning for In-Game Action Anticipation

et al. 2023

View full text Add to dashboard Cite

In real-time strategy (RTS) games, to defeat their opponents, players need to choose and implement the correct sequential actions. Because RTS games like StarCraft II are real-time, players have a very limited time to choose how to develop their strategy. In addition, players can only partially observe the parts of the map that they have explored. Therefore, unlike Chess or Go, players do not know what their opponents are doing. For these reasons, applying generally used artificial intelligence models to forecast sequential actions in RTS games is a challenge. To address this, we propose depthwise separable convolutionbased multimodal deep learning (DESEM) for forecasting sequential actions in the game StarCraft II. DESEM performs multimodal learning using high-dimensional frames and action labels simultaneously as inputs. We use a depthwise separable convolution as the backbone network for extracting features from highdimensional frames. In addition, we propose a weighted loss function to resolve class imbalances. We use 1,978 StarCraft II replays where the Terrans win in a Terran vs. Protoss game. The experimental results show that the proposed depthwise separable convolution is superior to the conventional convolution. Furthermore, we demonstrate that multimodal learning and the weighted loss function contribute significantly to improving forecasting performance.

show abstract

Section: Proposed Methods a Data Acquisition And Automated Labelingmentioning

confidence: 99%

DESEM: Depthwise Separable Convolution-Based Multimodal Deep Learning for In-Game Action Anticipation

et al. 2023

View full text Add to dashboard Cite

show abstract

“…We have implemented Keras-Retinanet on the simulated aerial data generated using MAVS [9]. RetinaNet is one of the best single stage object detection models that has proven to work well with dense and small-scale objects.…”

Section: Object Detection and Modellingmentioning

confidence: 99%

Object detection in synthetic aerial imagery using deep learning

Dabbiru

Goodin

Carruth

et al. 2023

Autonomous Systems: Sensors, Processing and Security for Ground, Air, Sea, and Space Vehicles and Infrastructure 2023

View full text Add to dashboard Cite

Object detection in aerial images is a challenging task as some objects are only a few pixels wide, some objects are occluded, and some are in shade. With the cost of drones decreasing, there is a surge in the amount of aerial data, so it will be useful if models can extract valuable features from the aerial data. Convolutional neural networks (CNN) are a useful tool for object detection and machine learning applications. However, machine learning requires labeled data to train and test the CNN models. In this work, we used a simulator to automatically generate labeled synthetic aerial imagery to use in the training and testing of machine learning algorithms. The synthetic aerial data used in this work was developed using a physics-based software tool called Mississippi State University Autonomous Vehicle Simulator (MAVS). We generated a dataset of 871 aerial images of 640x480 resolution and implemented Keras-RetinaNet framework with ResNet 50 as backbone for object detection. Keras-RetinaNet is one of the popular object detection models to be used with aerial imagery. As a preliminary task, we detected buildings in the synthetic aerial imagery and our results show a high mAP (mean Average Precision) accuracy of 77.99% using the state-of-the-art RetinaNet model.

show abstract

“…7 The MAVS has been used to study machine learning in a variety of applications using both camera and lidar data. [8][9][10] In addition to MAVS, the Unreal Engine version 4 (UE4) was used to generate images. 11 In order to get a wide range of mesh fidelities, several different representations of tree meshes were used in both simulators.…”

Section: Generating Synthetic Imagesmentioning

confidence: 99%

Fidelity requirements for simulating sensor performance in autonomous ground vehicles

Goodin

Carruth

Dabbiru

et al. 2023

Synthetic Data for Artificial Intelligence and Machine Learning: Tools, Techniques, and Applications

View full text Add to dashboard Cite

Simulation has become an important enabler in the development and testing of autonomous ground vehicles (AGV), with simulation being used both to generate training data for AI/ML-based segmentation and classification algorithms and to enable in-the-loop testing of the AGV systems that use those algorithms. Furthermore, digital twins of physical test areas provide a safe, repeatable way to conduct critical safety and performance testing of these AI/ML algorithms and their performance on AGV systems. For both these digital twins and the sensor models that use them to generate synthetic data, it is important to understand the relationship between the fidelity of the scene/model and the accuracy of the resulting synthetic sensor data. This work presents a quantitative evaluation of the relationship between digital scene fidelity, sensor model fidelity, and the quality of the resulting synthetic sensor data, with a focus on camera data typically used on AGV to enable autonomous navigation.

show abstract

Training Artificial Intelligence Algorithms with Automatically Labelled UAV Data from Physics-Based Simulation Software

Cited by 10 publications

References 27 publications

DESEM: Depthwise Separable Convolution-Based Multimodal Deep Learning for In-Game Action Anticipation

DESEM: Depthwise Separable Convolution-Based Multimodal Deep Learning for In-Game Action Anticipation

Object detection in synthetic aerial imagery using deep learning

Fidelity requirements for simulating sensor performance in autonomous ground vehicles

Contact Info

Product

Resources

About