Effects of sample size on the performance of species distribution models

Abstract

A wide range of modelling algorithms is used by ecologists, conservation practitioners, and others to predict species ranges from point locality data. Unfortunately, the amount of data available is limited for many taxa and regions, making it essential to quantify the sensitivity of these algorithms to sample size. This is the first study to address this need by rigorously evaluating a broad suite of algorithms with independent presence–absence data from multiple species and regions. We evaluated predictions from 12 algorithms for 46 species (from six different regions of the world) at three sample sizes (100, 30, and 10 records). We used data from natural history collections to run the models, and evaluated the quality of model predictions with area under the receiver operating characteristic curve (AUC). With decreasing sample size, model accuracy decreased and variability increased across species and between models. Novel modelling methods that incorporate both interactions between predictor variables and complex response shapes (i.e. GBM, MARS‐INT, BRUTO) performed better than most methods at large sample sizes but not at the smallest sample sizes. Other algorithms were much less sensitive to sample size, including an algorithm based on maximum entropy (MAXENT) that had among the best predictive power across all sample sizes. Relative to other algorithms, a distance metric algorithm (DOMAIN) and a genetic algorithm (OM‐GARP) had intermediate performance at the largest sample size and among the best performance at the lowest sample size. No algorithm predicted consistently well with small sample size (n < 30) and this should encourage highly conservative use of predictions based on small sample size and restrict their use to exploratory modelling.

Optimizing resiliency of reserve networks to climate change: multispecies conservation planning in the Pacific Northwest, USA

“…Given these limitations, we found, as have previous studies, that the MAXENT method provides a robust means of developing niche models from nonsystematic presence-only data across a range of sample sizes Hernandez et al, 2006;Pearson et al, 2007;Wisz et al, 2008). Our suite of species generally showed a strong association with older forest.…”

“…MAXENT was designed for use with presence-only data, and performed well in comparison with 15 alternate methods on a wide variety of taxa in diverse regions . It has shown especially robust performance in comparison to alternate methods at small sample sizes (e.g., 10 records as here; Hernandez et al, 2006;Pearson et al, 2007;Wisz et al, 2008). MAXENT uses a 'maximum entropy' approach that compares presence locations to a random subset of $ 10 000 background or available locations (Phillips et al, 2006).…”

“…We then limited analysis to the owl and the 130 localized species with 10 or more records remaining after thinning ( Fig. 2) (Kremen et al, 2008;Wisz et al, 2008). The 130 localized species comprised 75 fungi, 21 lichen, 10 bryophytes, eight vascular plants, 12 mollusks, two amphibians, a mammal, and a bird (Table S1).…”

An overview of recent remote sensing and GIS based research in ecological informatics

“…Modelling activities are inevitably limited by the quantity and quality of the data sets used (Lobo, 2007;Wisz et al, 2008). Unfortunately, ecological data are often characterised by a large degree of inherent uncertainty and error.…”

“…For example, in addition to issues connected with the data set size (Strayer, 1999;Stockwell and Peterson, 2002;Wisz et al, 2008) the partitioning of cases between presence and absence cases is important. For example, the use of data sets that are greatly imbalanced in terms of the proportion of presences and of absences can be problematic (Real et al, 2006).…”