The STONE Curve: A ROC‐Derived Model Performance Assessment Tool

Abstract

A new model validation and performance assessment tool is introduced, the sliding threshold of observation for numeric evaluation (STONE) curve. It is based on the relative operating characteristic (ROC) curve technique, but instead of sorting all observations in a categorical classification, the STONE tool uses the continuous nature of the observations. Rather than defining events in the observations and then sliding the threshold only in the classifier/model data set, the threshold is changed simultaneously for both the observational and model values, with the same threshold value for both data and model. This is only possible if the observations are continuous and the model output is in the same units and scale as the observations, that is, the model is trying to exactly reproduce the data. The STONE curve has several similarities with the ROC curve—plotting probability of detection against probability of false detection, ranging from the (1,1) corner for low thresholds to the (0,0) corner for high thresholds, and values above the zero‐intercept unity‐slope line indicating better than random predictive ability. The main difference is that the STONE curve can be nonmonotonic, doubling back in both the x and y directions. These ripples reveal asymmetries in the data‐model value pairs. This new technique is applied to modeling output of a common geomagnetic activity index as well as energetic electron fluxes in the Earth's inner magnetosphere. It is not limited to space physics applications but can be used for any scientific or engineering field where numerical models are used to reproduce observations.

Analysis of Features in a Sliding Threshold of Observation for Numeric Evaluation (STONE) Curve

“…As presented in Liemohn et al (2020), this comparison yields both a left-right wiggle as well as an up-down ripple in the STONE curve. The wiggle occurs when POFD exhibits an increase with increasing threshold, seen in the red curve of Figure 13c as occurring between log-flux threshold settings of 4.0 and 4.7.…”

“…As an application of these relationships between scatterplot and STONE curve features, let us use the same example as in Liemohn et al (2020), specifically the prediction of energetic electron observations (40 keV energy channel) from the geosynchronous orbiting environmental satellites (GOESs) spacecraft (Rowland & Weigel, 2012) to real-time output from the IMPTAM. IMPTAM has been running in real time for nearly a decade (Ganushkina et al, 2015), and these two particular number sets were originally compared by Ganushkina et al (2019).…”

“…IMPTAM has been running in real time for nearly a decade (Ganushkina et al, 2015), and these two particular number sets were originally compared by Ganushkina et al (2019). The time period is from September 2013 to March 2015 and, as with Liemohn et al (2020), the magnetic local time (MLT) of the comparison will be restricted to the dawn sector (specifically, the 03-09 MLT range), when the model performs the best of any MLT sector.…”

“…In this particular case, there is no need to keep the observed event identification threshold constant; it can be swept along with the model event identification threshold. Such a technique was conducted by Liemohn et al (2020) to introduce the analysis method they called the sliding threshold of observation for numeric evaluation (STONE) curve. The STONE curve is like the ROC curve in that it is a plot of POD versus POFD, but the underlying contingency tables for each point on the curve are created by sliding both event identification thresholds simultaneously.…”

“…This leads to the misses and false alarm cell counts increasing and decreasing throughout the threshold sweep, possibly resulting in times where the POD or POFD metrics temporarily increase. Liemohn et al (2020) showed two space weather examples of the usage of the STONE curve. The resulting nonmonotonicities were qualitatively interpreted as intervals when clusters of points were quite far from the "ideal fit" diagonal line through the data-model scatterplot.…”