Optimal estimator for assessing landslide model performance

Abstract

Abstract. The commonly used success rate (SR) in evaluating cell-based landslide model performance is based on the ratio of successfully predicted landslide sites over total actual landslide sites without considering the performance in predicting stable cells. We proposed a modified SR (MSR), in which the performance of stable cell prediction is included. The advantage of MSR is to avoid over-and under-prediction while upholding the stable sensitivity throughout all simulated cases. Stochastic analyses are conducted by using artificial landslide maps and simulations with a full range of performances (from worst to perfect) in both stable and unstable cell predictions. Stochastic analyses reveal mathematical responses of estimators to various model results in calculating performance. The Kappa method, which is commonly used for satellite image analysis, is improper for landslide modeling giving inconsistent performance when landslide coverage changes. To examine differences among SR and MSR in real model application, we applied the SHALSTAB model onto a mountainous watershed in Taiwan. Case study shows that stable and unstable cell predictions are inter-exclusive in SHALSTAB model. The optimal estimator should compromise landslide over-and under-prediction. According to our 4000 simulations, the best simulation generated by MSR projects 83 hits over 131 actual landslide sites while the unstable cells cover only 16% of the studied watershed. By contrast, despite the fact that the best simulation deduced from SR projects 120 hits over 131 actual landslide sites, this high performance is only obtained when unstable cells cover an incredibly high landslide cover (∼75%) of the entire watershed exhibiting a significant landslide over-prediction.

The influence of grid resolution on the prediction of natural and road-related shallow landslides

“…This result is consistent with the general tendency of several process-based models to overestimate the areas predicted as unconditionally un- (Huang and Kao, 2006). Montgomery and Dietrich (1994) reported percentages of unconditionally unstable areas ranging between 1 and 13 %, clearly exceeding the observed landslide area, for the first application of SHALSTAB.…”

Landslide and debris flow susceptibility zonation using TRIGRS for the 2011 Seoul landslide event

“…An ideal landslide assessment model simultaneously maximizes the agreement between known and predicted locations of landslides, and minimizes predicted unstable area to give useful information for prediction. In order to overcome the disadvantages and limitations of such comparisons, various indices have been proposed: SR and MSR stand for success rate and modified success rate (Huang and Kao, 2006); ROC stands for receiver operating characteristic using confusion matrix Montrasio, 2011;Raia et al, 2013); SI and EI stand for success index and error index (Sorbino et al, 2010); SC and LP stand for scar concentration and landslide potential (Vieira et al, 2010); POD, FAR and CSI stand for probability of detection, false alarm ratio and critical success index (Liao et al, 2011); and the D index (Liu and Wu, 2008).…”

Developing an Accessible Landslide Susceptibility Model Using Open-Source Resources

“…Most previous studies used the coincidence of pixels between simulated and historical landslides to verify the performance of their models [40,41]. However, to overcome the limitations of previously simulated models, various indices have been suggested, such as the success rate and the modified success rate [42], the D index [30], the receiver operating characteristic [43,44], the success and error indices [45], the scar concentration and landslide potential [14], the probability of detection, the false alarm ratio, and the critical success index [46]. The results of this study can be utilized in landslide hazard assessment and urban planning studies [47][48][49] as the cost-effective and efficient data and software application approach.…”