How Response Designs and Class Proportions Affect the Accuracy of Validation Data

Abstract

Reference data collected to validate land cover maps are generally considered free of errors. In practice, however, they contain errors despite all efforts to minimise them. These errors then propagate up to the accuracy assessment stage and impact the validation results. For photo-interpreted reference data, the three most widely studied sources of error are systematic incorrect labelling, vigilance drops, and demographic factors. How internal estimation errors, i.e., errors intrinsic to the response design, affect the accuracy of reference data is far less understood. In this paper, we analysed the impact of estimation errors for two types of legends (binary and multi-class) as well as for two common response designs (point-based and partition-based) with a range of sub-sample sizes. We showed that the accuracy of response designs depends on the class proportions within the sampling units, with complex landscapes being more prone to errors. As a result, response designs where the number of sub-samples are fixed are inefficient, and the labels of reference data sets have inconsistent confidence levels. In order to control estimation errors, to guarantee high accuracy standards of validation data, and to minimise data collection efforts, we proposed to rely on confidence intervals of the photo-interpreted data to define how many sub-samples should be labelled. In practice, sub-samples are iteratively selected and labelled until the estimated class proportions reach the desired level of confidence. As a result, less effort is spent on labelling obvious cases and the spared effort can be allocated to more complex cases, thus leading to an increased reliability of reference data and in turn better accuracy assessment. Across our study site, we demonstrated that such an approach could reduce the labelling effort by 50% to 75%, with greater gains in homogeneous landscapes. We contend that adopting this optimisation approach will not only increase the efficiency of reference data collection but will also help deliver reliable accuracy estimates to the user community.

First 1-M Resolution Land Cover Map Labeling the Overlap in the 3rd Dimension: The 2018 Map for Wallonia

“…The very high spatial resolution of this map makes it relevant for a land cover typology that neglects mixed pixels. Indeed, even if mixed pixels do occur at any scale, their relative proportion at 1 m 2 is very small [24]. Because most of the sources had a better spatial resolution than the final data, a majority rule has been used to agglomerate these sources at 1 m resolution.…”

About the Pitfall of Erroneous Validation Data in the Estimation of Confusion Matrices

“…The sources of errors in the collection of ground reference data are well documented, and some recent studies aimed at assessing their contribution depending on whether the errors are thematic or positional. Thematic errors consist in assigning the wrong label to a sample unit because of erroneous classification, (uncertainties of the response design [7], temporal mismatch, transitional classes, class interpretation errors, careless error [8]). Positional errors (also called geolocation errors) may result in incorrect matching of reference and map labels because the geographic position of the sampling unit is shifted with respect to the map (mislocation of testing sites, mislocation of the map or uncertain definition of boundaries) [2,9,10].…”