A large‐area, spatially continuous assessment of land cover map error and its impact on downstream analyses

Abstract

Land cover maps increasingly underlie research into socioeconomic and environmental patterns and processes, including global change. It is known that map errors impact our understanding of these phenomena, but quantifying these impacts is difficult because many areas lack adequate reference data. We used a highly accurate, high-resolution map of South African cropland to assess (1) the magnitude of error in several current generation land cover maps, and (2) how these errors propagate in downstream studies. We first quantified pixel-wise errors in the cropland classes of four widely used land cover maps at resolutions ranging from 1 to 100 km, and then calculated errors in several representative "downstream" (map-based) analyses, including assessments of vegetative carbon stocks, evapotranspiration, crop production, and household food security. We also evaluated maps' spatial accuracy based on how precisely they could be used to locate specific landscape features. We found that cropland maps can have substantial biases and poor accuracy at all resolutions (e.g., at 1 km resolution, up to $ 45% underestimates of cropland (bias) and nearly 50% mean absolute error (MAE, describing accuracy); at 100 km, up to 15% underestimates and nearly 20% MAE). National-scale maps derived from higherresolution imagery were most accurate, followed by multi-map fusion products.Constraining mapped values to match survey statistics may be effective at minimizing bias (provided the statistics are accurate). Errors in downstream analyses could be substantially amplified or muted, depending on the values ascribed to croplandadjacent covers (e.g., with forest as adjacent cover, carbon map error was 200%-500% greater than in input cropland maps, but $ 40% less for sparse cover types).The average locational error was 6 km (600%). These findings provide deeper insight into the causes and potential consequences of land cover map error, and suggest several recommendations for land cover map users. K E Y W O R D Sagent-based model, agriculture, bias, carbon, crop yield, evapotranspiration, land cover, remote sensing

Accounting for Training Data Error in Machine Learning Applied to Earth Observations

“…While TD is more visible in the context of LCLU and other categorical mapping projects, the need for rigorous, well-documented TD is also critically important for continuous variable applications in Earth System Sciences (e.g., hydrological research [228]). If adopted within the peer-reviewed literature, the standards we propose for TD treatment may improve confidence in scientific findings drawn from map-based research, which can otherwise be confounded by poorly quantified map errors [33,57]. Figure S2: Schematic of product selection using the Triple Collocation approach.…”

“…Less information is available regarding the ways in which TD error may propagate beyond the map it initially creates. Initial research by Estes et al [33] examined how error propagates from a primary land-cover map into subsequent derived products. This work used a high-quality reference cropland map to quantify the errors in 1 km cropland fractions derived from existing land cover datasets and measured how these errors propagated in several map-based analyses drawing on cropland fractions for inputs.…”

“…In the current data-rich era, the impact of training data (TD) quality and quantity on map accuracy is even more relevant, especially for maps generated by data-hungry ML algorithms [27][28][29][30][31][32]. Errors in these products also impact the veracity of any downstream products into which they are ingested [33]. While progress in algorithmic performance continues apace, standards regarding the collection and use of TD remain uncoordinated across researchers [34].…”

Change versus stability: are protected areas particularly pressured by global land cover change?

“…As for many land cover products at global scale, change detection can be underestimated under the current analysis (Jamali et al 2015;Tsendbazar et al 2015;Liu et al 2018). This underestimation is related to the current limitations of land cover change detection at broader scales (Lunetta et al 2004(Lunetta et al , 2006Fuchs et al 2015;Estes et al 2018), but also with the inability to monitor and map land use change, decreasing the capacity to determine the impacts on biodiversity and ecosystems more effectively.…”

A cultivated planet in 2010: 2. the global gridded agricultural production maps

“…Agricultural production systems are constantly changing, hence the spatial characterization at one year might be notably different than that on another. However, a lot of recent 70 agricultural and environmental assessments were still based on those maps produced decades ago (Deutsch et al, 2018;Nanni et al, 2019;Estes et al, 2018;Prestele et al, 2018;Erb et al, 2018;Porwollik et al, 2019;Yu et al, 2017b).…”