Testing the ability of species distribution models to infer variable importance

Abstract

Models of species’ distributions and niches are frequently used to infer the importance of range‐ and niche‐defining variables. However, the degree to which these models can reliably identify important variables and quantify their influence remains unknown. Here we use a series of simulations to explore how well models can 1) discriminate between variables with different influence and 2) calibrate the magnitude of influence relative to an ‘omniscient’ model. To quantify variable importance, we trained generalized additive models (GAMs), Maxent and boosted regression trees (BRTs) on simulated data and tested their sensitivity to permutations in each predictor. Importance was inferred by calculating the correlation between permuted and unpermuted predictions, and by comparing predictive accuracy of permuted and unpermuted predictions using AUC and the continuous Boyce index. In scenarios with one influential and one uninfluential variable, models failed to discriminate reliably between variables when training occurrences were < 8–64, prevalence was > 0.5, spatial extent was small, environmental data had coarse resolution and spatial autocorrelation was low, or when pairwise correlation between environmental variables was |r| > 0.7. When two variables influenced the distribution equally, importance was underestimated when species had narrow or intermediate niche breadth. Interactions between variables in how they shaped the niche did not affect inferences about their importance. When variables acted unequally, the effect of the stronger variable was overestimated. GAMs and Maxent discriminated between variables more reliably than BRTs, but no algorithm was consistently well‐calibrated vis‐à‐vis the omniscient model. Algorithm‐specific measures of importance like Maxent's change‐in‐gain metric were less robust than the permutation test. Overall, high predictive accuracy did not connote robust inferential capacity. As a result, requirements for reliably measuring variable importance are likely more stringent than for creating models with high predictive accuracy.

Toward a Monte Carlo approach to selecting climate variables in MaxEnt

“…There is heightened awareness of the significance of dimensionality in understanding environmental spaces and the importance of variable selection in modeling those spaces [ 23 , 34 , 80 ]. This awareness is accompanied by a recognition that logistic difficulties often preclude examining large numbers of variables [ 62 ]. This has led to a search for alternative means of variable selection and calls for process automation [ 22 , 23 , 62 , 78 , 81 ].…”

“…This awareness is accompanied by a recognition that logistic difficulties often preclude examining large numbers of variables [ 62 ]. This has led to a search for alternative means of variable selection and calls for process automation [ 22 , 23 , 62 , 78 , 81 ]. A comprehensive review of these approaches is beyond the scope of this paper; however, it is worth nothing that even among the most recent work in this area, many of the solutions put forward—such as manual prescreening for collinear variables, greater use of biological insight in variable selection, broader use of memory-resident machine language-based analysis software, etc.—do not, in general, scale well.…”

“…MaxEnt provides three algorithm-specific indicators of variable importance: percent contribution, permutation importance, and change-in-gain based on jackknife analysis of individual variables [ 3 , 62 , 65 ]. No single measure is sufficient to identify which variables are best for producing a final model [ 23 ]; however, for screening purposes and to simplify comparisons in this initial evaluation, we used permutation importance as our sole indicator of variable importance.…”

“…In addition to GDAL, our computing environment comprised MaxEnt Version 3. One of the most common uses for ecological niche models is to identify important variables [62,63]. In this study, we used MaxEnt in two different ways to find the six most influential range-and niche-defining bioclim variables for Cassin's Sparrow.…”

Untitled

“…Through these two types of evaluations, we identified the most important variables within the ones evaluated in this exercise. We acknowledge that this exercise can identify the variables that lead to models with the highest predictive accuracy, but not necessarily identify the actual environmental tolerances of the organisms [33].…”