Performance metrics for the assessment of satellite data products: an ocean color case study

Abstract

Performance assessment of ocean color satellite data has generally relied on statistical metrics chosen for their common usage and the rationale for selecting certain metrics is infrequently explained. Commonly reported statistics based on mean squared errors, such as the coefficient of determination (r), root mean square error, and regression slopes, are most appropriate for Gaussian distributions without outliers and, therefore, are often not ideal for ocean color algorithm performance assessment, which is often limited by sample availability. In contrast, metrics based on simple deviations, such as bias and mean absolute error, as well as pair-wise comparisons, often provide more robust and straightforward quantities for evaluating ocean color algorithms with non-Gaussian distributions and outliers. This study uses a SeaWiFS chlorophyll-a validation data set to demonstrate a framework for satellite data product assessment and recommends a multi-metric and user-dependent approach that can be applied within science, modeling, and resource management communities.

Phytoplankton Size Structure in Association with Mesoscale Eddies off Central-Southern Chile: The Satellite Application of a Phytoplankton Size-Class Model

“…The latter value is, however, higher than that obtained by Ward [46]. In applying the back-transformation procedure recommended by Seegers et al [67], we considered the metrics MdAE and bias for comparison (Table 3). In the interpretation of these metrics (MdAE(t) and δ(t)), values closer to unity indicate lower relative errors and bias, with biases higher (lower) than unity implying that the model overestimates (underestimates) in situ measurements [67].…”

“…Finally, the Pearson linear correlation coefficient (r), root mean square (RMS) error, and bias (δ) were calculated in log 10 space as statistical metrics for model and satellite model validation following Brewin et al [48], in order to compare them with those from previous studies. Recently Seegers et al [67] have queried the use of RMS as an error metric in the case of Chl-a data and have instead recommended the use of the mean/median absolute error (MAE and MdAE) and the median bias (δ). In addition, these authors recommend that these metrics be back-transformed from the log 10 space for interpretation.…”

“…In addition, these authors recommend that these metrics be back-transformed from the log 10 space for interpretation. For this reason, we have also included the metrics and procedures suggested by Seegers et al [67].…”

“…In applying the back-transformation procedure recommended by Seegers et al [67], we considered the metrics MdAE and bias for comparison (Table 3). In the interpretation of these metrics (MdAE(t) and δ(t)), values closer to unity indicate lower relative errors and bias, with biases higher (lower) than unity implying that the model overestimates (underestimates) in situ measurements [67]. In terms of the MdAE(t) results, the highest uncertainties were associated with the micro-(~69%) and picoplankton (~97%) size classes in comparison with the other PSC (<30%); bias values were close to unity, except for picoplankton (Table 3).…”

“…In coastal upwelling regions, such as in this study, total Chl-a values reach up to 50 mg m −3 [85], implying that the range included here is very narrow. The accuracy in the estimates for all PSC could be also related to (i) deviations in the relationship between the PSC and total Chl-a previously reported for optically complex waters, often found in coastal systems [48]; (ii) the spatial scale when comparing 4 km satellite pixels with specific in situ value obtained from~250-300 mL of water, involving an additional sub-pixel variability in PSC estimates [48]; (iii) the estimation of total Chl-a concentrations from the ocean colour algorithms which are based on an assumed relationship between the total Chl-a and the remote sensing reflectance, both of which could vary according to cell size [46,67]; and (iv) the optical depth [43,50], an aspect that could be explored in future works.…”