Testing prediction accuracy in short-term ecological studies

Wood, Connor M.; Loman, Zachary G.; McKinney, Shawn T.; Loftin, Cynthia S.

doi:10.1016/j.baae.2020.01.003

Cited by 7 publications

(8 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Information criteria are often favored by ecologists because they are considered effective measurements of model parsimony (Aho et al 2014). However, based on results from our study and other recent evaluations (Clark et al 2020, Wood et al 2020), we suggest that model selection using information criteria does not always identify the most parsimonious model(s) either. There remains considerable debate about whether ecologists should be using complex or parsimonious models to analyze data (e.g., Evans et al 2013, Coelho et al 2019), but the justification for using complex or parsimonious models largely depends on whether the analysis is structured around mechanistic or correlative models (Coelho et al 2019).…”

Section: Discussionsupporting

confidence: 46%

“…This assumption is based on the fact that numerous information criteria like the AIC, DIC, and WAIC, can be interpreted as measures of prediction error for new data points (Gelman et al 2014; e.g., the AIC is considered asymptotically equivalent to leave‐one‐out cross‐validation; Stone 1977). However, of the few studies that have empirically tested this assumption using population models, most have found that information criteria are not reliable substitutions for empirical measures of predictive error (Link and Sauer 2016, Link et al 2017, Clark et al 2020, Wood et al 2020). Our analysis is consistent with these previous studies.…”

Section: Discussionmentioning

confidence: 99%

“…If the latter, like in our analysis, then ecologists should strive for model parsimony. Indeed, using a variety of validation approaches, researchers have found simpler models (or models of intermediate complexity) generally perform as well as, or better than, more complex models in predicting out‐of‐sample data (e.g., Ward et al 2014, Tredennick et al 2016, Chevalier and Knape 2020, Clark et al 2020, Wood et al 2020). But approaches such as borrowing information across sites (when possible) appear to improve the predictive ability of complex models, presumably by reducing over‐fitting (Chevalier and Knape 2020).…”

Section: Discussionmentioning

confidence: 99%

See 2 more Smart Citations

Ecological forecasts reveal limitations of common model selection methods: predicting changes in beaver colony densities

Johnson‐Bice

Ferguson

Erb

et al. 2020

Ecological Applications

View full text Add to dashboard Cite

Over the past two decades, there have been numerous calls to make ecology a more predictive science through direct empirical assessments of ecological models and predictions. While the widespread use of model selection using information criteria has pushed ecology toward placing a higher emphasis on prediction, few attempts have been made to validate the ability of information criteria to correctly identify the most parsimonious model with the greatest predictive accuracy. Here, we used an ecological forecasting framework to test the ability of information criteria to accurately predict the relative contribution of density dependence and density-independent factors (forage availability, harvest, weather, wolf [Canis lupus] density) on inter-annual fluctuations in beaver (Castor canadensis) colony densities. We modeled changes in colony densities using a discrete-time Gompertz model, and assessed the performance of four models using information criteria values: density-independent models with (1) and without (2) environmental covariates; and density-dependent models with (3) and without (4) environmental covariates. We then evaluated the forecasting accuracy of each model by withholding the final one-third of observations from each population and compared observed vs. predicted densities. Information criteria and our forecasting accuracy metrics both provided strong evidence of compensatory density dependence in the annual dynamics of beaver colony densities. However, despite strong within-sample performance by the most complex model (density-dependent with covariates) as determined using information criteria, hindcasts of colony densities revealed that the much simpler density-dependent model without covariates performed nearly as well predicting out-of-sample colony densities. The hindcast results indicated that the complex model over-fit our data, suggesting that parameters identified by information criteria as important predictor variables are only marginally valuable for predicting landscapescale beaver colony dynamics. Our study demonstrates the importance of evaluating ecological models and predictions with long-term data and revealed how a known limitation of information criteria (over-fitting of complex models) can affect our interpretation of ecological dynamics. While incorporating knowledge of the factors that influence animal population dynamics can improve population forecasts, we suggest that comparing forecast performance metrics can likewise improve our knowledge of the factors driving population dynamics.

show abstract

Section: Discussionsupporting

confidence: 46%

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Ecological forecasts reveal limitations of common model selection methods: predicting changes in beaver colony densities

Johnson‐Bice

Ferguson

Erb

et al. 2020

Ecological Applications

View full text Add to dashboard Cite

show abstract

“…Yet, comparing models across studies for applied decision‐making occurs infrequently (though see Lewis, Rose, et al, 2021). We need the best models of appropriate complexity (Aho et al, 2014; Anderson et al, 2000; Horne & Garton, 2006; Wood et al, 2020) for each application (Dietze, 2017), even for cases where we do not have the resources of unified global efforts (e.g., IPCC: Masson‐Delmotte et al, 2021). Furthermore, the generality of models can be determined by how well they predict in many contexts.…”

Section: Introductionmentioning

confidence: 99%

PERFICT: A Re‐imagined foundation for predictive ecology

et al. 2022

View full text Add to dashboard Cite

Making predictions from ecological models—and comparing them to data—offers a coherent approach to evaluate model quality, regardless of model complexity or modelling paradigm. To date, our ability to use predictions for developing, validating, updating, integrating and applying models across scientific disciplines while influencing management decisions, policies, and the public has been hampered by disparate perspectives on prediction and inadequately integrated approaches. We present an updated foundation for Predictive Ecology based on seven principles applied to ecological modelling: make frequent Predictions, Evaluate models, make models Reusable, Freely accessible and Interoperable, built within Continuous workflows that are routinely Tested (PERFICT). We outline some benefits of working with these principles: accelerating science; linking with data science; and improving science‐policy integration.

show abstract

“…While shorter-term studies (i.e. those where data collection occurs for less than ~5 years) that coincide with length of typical grant cycles and graduate programs are still the norm, these human constraints do not necessarily capture the ecological phenomena they seek to measure, particularly their temporal dependencies (Hastings, 2004;Wood et al, 2020). This unfortunate mismatch of scales has the potential to limit our understanding of ecological trajectories-that is, the direction a system is going through time, and can undermine our efforts towards a predictive ecology (Evans et al, 2012).…”

Section: Introductionmentioning

confidence: 99%

The Broken Window: An algorithm for quantifying and characterizing misleading trajectories in ecological processes

Bahlai

White

Perrone

et al. 2020

Preprint

View full text Add to dashboard Cite

AbstractA fundamental problem in ecology is understanding how to scale discoveries: from patterns we observe in the lab or the plot to the field or the region or bridging between short term observations to long term trends. At the core of these issues is the concept of trajectory—that is, when can we have reasonable assurance that we know where a system is going? In this paper, we describe a non-random resampling method to directly address the temporal aspects of scaling ecological observations by leveraging existing data. Findings from long-term research sites have been hugely influential in ecology because of their unprecedented longitudinal perspective, yet short-term studies more consistent with typical grant cycles and graduate programs are still the norm.We directly address bridging the gap between the short-term and the long-term by developing an automated, systematic resampling approach: in short, we repeatedly ‘sample’ moving windows of data from existing long-term time series, and analyze these sampled data as if they represented the entire dataset. We then compile typical statistics used to describe the relationship in the sampled data, through repeated samplings, and then use these derived data to gain insights to the questions: 1) how often are the trends observed in short-term data misleading, and 2) can we use characteristics of these trends to predict our likelihood of being misled? We develop a systematic resampling approach, the ‘bad-breakup’ algorithm, and illustrate its utility with a case study of firefly observations produced at the Kellogg Biological Station Long-Term Ecological Research Site (KBS LTER). Through a variety of visualizations, summary statistics, and downstream analyses, we provide a standardized approach to evaluating the trajectory of a system, the amount of observation required to find a meaningful trajectory in similar systems, and a means of evaluating our confidence in our conclusions.

show abstract

Testing prediction accuracy in short-term ecological studies

Cited by 7 publications

References 28 publications

Ecological forecasts reveal limitations of common model selection methods: predicting changes in beaver colony densities

Ecological forecasts reveal limitations of common model selection methods: predicting changes in beaver colony densities

PERFICT: A Re‐imagined foundation for predictive ecology

The Broken Window: An algorithm for quantifying and characterizing misleading trajectories in ecological processes

Contact Info

Product

Resources

About