Deep Learning Classification of 2D Orthomosaic Images and 3D Point Clouds for Post-Event Structural Damage Assessment

Abstract

Efficient and rapid data collection techniques are necessary to obtain transitory information in the aftermath of natural hazards, which is not only useful for post-event management and planning, but also for post-event structural damage assessment. Aerial imaging from unpiloted (gender-neutral, but also known as unmanned) aerial systems (UASs) or drones permits highly detailed site characterization, in particular in the aftermath of extreme events with minimal ground support, to document current conditions of the region of interest. However, aerial imaging results in a massive amount of data in the form of two-dimensional (2D) orthomosaic images and three-dimensional (3D) point clouds. Both types of datasets require effective and efficient data processing workflows to identify various damage states of structures. This manuscript aims to introduce two deep learning models based on both 2D and 3D convolutional neural networks to process the orthomosaic images and point clouds, for post windstorm classification. In detail, 2D convolutional neural networks (2D CNN) are developed based on transfer learning from two well-known networks AlexNet and VGGNet. In contrast, a 3D fully convolutional network (3DFCN) with skip connections was developed and trained based on the available point cloud data. Within this study, the datasets were created based on data from the aftermath of Hurricanes Harvey (Texas) and Maria (Puerto Rico). The developed 2DCNN and 3DFCN models were compared quantitatively based on the performance measures, and it was observed that the 3DFCN was more robust in detecting the various classes. This demonstrates the value and importance of 3D datasets, particularly the depth information, to distinguish between instances that represent different damage states in structures.

Data-Driven Approaches for Tornado Damage Estimation with Unpiloted Aerial Systems

“…From 2D images to 3D point clouds, DNNs have also been applied to estimate tornado damage using point clouds generated by SfM. 3D fully convolutional networks (FCNs) were used to segment 3D voxels rasterized from post-wind point clouds [27,28]. 3D voxel segmentation requires values for every voxel in 3D grids, including empty spaces, which were labeled as neutral in their work [27,28].…”

“…3D fully convolutional networks (FCNs) were used to segment 3D voxels rasterized from post-wind point clouds [27,28]. 3D voxel segmentation requires values for every voxel in 3D grids, including empty spaces, which were labeled as neutral in their work [27,28]. This additional information could have affected the accuracy of their results by adding to the imbalanced training dataset problem [29].…”

Collapsed Building Detection Using 3D Point Clouds and Deep Learning

“…The inferred primary reason for the limited number of works is the absence of a large-scale airborne LiDAR dataset tailored to 3D building damage detection. Although an SfM-based point cloud dataset was developed and the performance of 3D voxel-based DNN was tested using the developed dataset in [27], the size of the dataset developed was relatively small. A large-scale dataset is a crucial and fundamental resource for developing advanced algorithms for targeted tasks, as well as for providing training and benchmarking data for such algorithms [28][29][30][31], which requires decent expertise and can be labor-intensive.…”

Grand Challenges in Earth Observation Using Unpiloted Airborne Systems

“…Advances intechnologyandavailabilityhaveboosted the number of applications in many sectors but especially in regions with local and regional dynamic features. Just a few innovative and successful examples of UAS-based monitoring are glacier monitoring for ice flow and mass wasting (Immerzeel et al, 2014;Kraaijenbrink et al, 2016), landslide dynamics monitoring and surface deformation (McKean, 2004;Niethammer et al, 2012;Lucieer et al, 2014;Giordan et al, 2020;Karantanellis et al, 2020), dune dynamics (Ruessink et al, 2018), flood risk mapping (Hashemi-Beni et al, 2018), night-time light monitoring as proxy for economic activity (Li et al, 2020), public health care and health-related services (Amukele et al, 2015;Scalea, 2020), and post-disaster damage assessment (Kerle et al, 2019;Liao et al, 2020).…”

“…Machine learning and advanced "Big Data" analysis methods have developed considerably over recent years. Methods such as Random Forests, Support Vector Machines, Convolution Neural Networks will contribute largely to the high-speed and possible automated analysis of the huge amount of drone data that can be collected during just a few field surveys (Khan and Al-Mulla, 2019;Liao et al, 2020).…”