Evaluating presence‐only species distribution models with discrimination accuracy is uninformative for many applications

Abstract

Aim Species distribution models are used across evolution, ecology, conservation and epidemiology to make critical decisions and study biological phenomena, often in cases where experimental approaches are intractable. Choices regarding optimal models, methods and data are typically made based on discrimination accuracy: a model's ability to predict subsets of species occurrence data that were withheld during model construction. However, empirical applications of these models often involve making biological inferences based on continuous estimates of relative habitat suitability as a function of environmental predictor variables. We term the reliability of these biological inferences ‘functional accuracy.’ We explore the link between discrimination accuracy and functional accuracy. Methods Using a simulation approach we investigate whether models that make good predictions of species distributions correctly infer the underlying relationship between environmental predictors and the suitability of habitat. Results We demonstrate that discrimination accuracy is only informative when models are simple and similar in structure to the true niche, or when data partitioning is geographically structured. However, the utility of discrimination accuracy for selecting models with high functional accuracy was low in all cases. Main conclusions These results suggest that many empirical studies and decisions are based on criteria that are unrelated to models’ usefulness for their intended purpose. We argue that empirical modelling studies need to place significantly more emphasis on biological insight into the plausibility of models, and that the current approach of maximizing discrimination accuracy at the expense of other considerations is detrimental to both the empirical and methodological literature in this active field. Finally, we argue that future development of the field must include an increased emphasis on simulation; methodological studies based on ability to predict withheld occurrence data may be largely uninformative about best practices for applications where interpretation of models relies on estimating ecological processes, and will unduly penalize more biologically informative modelling approaches.

Associations Between Habitat Quality And Body Size In The Carpathian Land SnailVestia turgida: Species Distribution Model Selection And Assessment Of Performance

“…However, by testing several SDM algorithms, we found that some of them performed better, others not so good, in terms of expounding this correlation. In other words, their functional accuracy (Warren et al, 2020) was different. Therefore, marginal R 2 , expressing the variance explained by the fixed terms in the regression models, was adopted as a measure of functional accuracy, and used to rank the SDMs accordingly.…”

“…Traditionally, determining environmental and climatic features that characterize the species' niche and are responsible for shaping their distribution would require laborious field measurements of key environmental variables in natural populations (Nakazato et al, 2010;Warren et al, 2020). Importantly, the use of SDMs has allowed to identify such driving factors, but SDM construction involves many decisions which may adversely affect model predictions, including the choice of modelling algorithms (Warren et al, 2020).…”

“…Traditionally, determining environmental and climatic features that characterize the species' niche and are responsible for shaping their distribution would require laborious field measurements of key environmental variables in natural populations (Nakazato et al, 2010;Warren et al, 2020). Importantly, the use of SDMs has allowed to identify such driving factors, but SDM construction involves many decisions which may adversely affect model predictions, including the choice of modelling algorithms (Warren et al, 2020). Choices regarding optimal models and methods are typically made based on discrimination accuracy, which only measures whether a model assigns higher suitability values to presence points than it does to background or absence points (Gurgel-Gonçalves et al, 2012), and conclusions have been made of the inability of current evaluation metrics to assess the biological significance of distribution models (Fourcade et al, 2018).…”

“…Yet, in the few studies that have explicitly tested the relationship between habitat quality and species traits, not always such relationship was found (Mammola et al, 2019). The problem could be that it is often not clear which measurable biological phenomena should be correlated with suitability estimates from SDMs, moreover when many of the measurable phenomena that are potentially related to suitability have not been quantified in detail and as such are merely unavailable for model validation (Warren et al, 2020).…”

Assessing the reliability of species distribution projections in climate change research

“…Manish and Pandit 2019), increasing the risk of detecting spurious relationships (Synes andOsborne 2011, Fourcade et al 2018). Worryingly, it has been shown that non-biologically relevant predictors can contribute to increase the predictive ability of the models (Fourcade et al 2018), so discrimination accuracy metrics may suggest a very good model while the relationships estimated do not have a biological meaning (Journé et al 2019, Warren et al 2020.…”