Effect of Label Noise on the Machine-Learned Classification of Earthquake Damage

Abstract

Automated classification of earthquake damage in remotely-sensed imagery using machine learning techniques depends on training data, or data examples that are labeled correctly by a human expert as containing damage or not. Mislabeled training data are a major source of classifier error due to the use of imprecise digital labeling tools and crowdsourced volunteers who are not adequately trained on or invested in the task. The spatial nature of remote sensing classification leads to the consistent mislabeling of classes that occur in close proximity to rubble, which is a major byproduct of earthquake damage in urban areas. In this study, we look at how mislabeled training data, or label noise, impact the quality of rubble classifiers operating on high-resolution remotely-sensed images. We first study how label noise dependent on geospatial proximity, or geospatial label noise, compares to standard random noise. Our study shows that classifiers that are robust to random noise are more susceptible to geospatial label noise. We then compare the effects of label noise on both pixel-and object-based remote sensing classification paradigms. While object-based classifiers are known to outperform their pixel-based counterparts, this study demonstrates that they are more susceptible to geospatial label noise. We also introduce a new labeling tool to enhance precision and image coverage. This work has important implications for the Sendai framework as autonomous damage classification will ensure rapid disaster assessment and contribute to the minimization of disaster risk.

An Entropic Optimal Transport loss for learning deep neural networks under label noise in remote sensing images

“…In remote sensing image analysis, the adverse effect of the label noise is not much studied in literature. The impact of label noise has been recently studied in (Frank et al, 2017;Pelletier et al, 2017a) with shallow classifiers. The feasibility of using online open street map (outdated or mislabeled ground truth) to obtain classification map with deep neural network was studied in (Kaiser et al, 2017), however they didn't consider directly addressing label noise as a specificity of the problem.…”

Novel Unsupervised Classification of Collapsed Buildings Using Satellite Imagery, Hazard Scenarios and Fragility Functions

“…Consequently, more robust and versatile calibration methods are needed. To this end, machine learning techniques have been successfully applied [16][17][18][19][20][21][22]. Calibration using supervised machine learning is performed based on training samples, that is a set of features for which the corresponding damage states are known in advance.…”

New Insights into Multiclass Damage Classification of Tsunami-Induced Building Damage from SAR Images

“…The quality and quantity of training data play a pivotal role in supervised machine learning. These training data can be obtained through visual inspection of high-resolution optical images [19]. Moreover, by replacing training samples with probabilistic information derived from the spatial distribution of the hazard, databases of previous disasters can be used for constructing a discriminant function [20].…”