Evaluating goodness-of-fit of animal movement models using lineups

Abstract

Models of animal movement are frequently fit to animal location data to understand how animals respond to and interact with local environmental features. Several open-source software packages are available for analyzing animal movements and can facilitate parameter estimation, yet there are relatively few methods available for evaluating model goodness-of-fit. We describe how a simple graphical technique, thelineup protocol, can be used to evaluate goodness-of-fit of integrated step-selection analyses and hidden Markov models, but the method can be applied much more broadly. We leverage the ability to simulate data from fitted models, and demonstrate the approach using both methods applied to fisher (Pekania pennanti) data. A variety of responses and movement metrics can be used to evaluate models, and the lineup protocol can be tailored to focus on specific model assumptions or movement features that are of primary interest. Although it is possible to evaluate goodness-of-fit using a formal hypothesis test, the method can also be used in a more exploratory fashion (e.g., to visualize variability in model behavior across stochastic simulations or identify areas where the model could be improved). We provide coded examples and two vignettes to demonstrate the flexibility of the approach and encourage movement ecologists to consider how their models will be applied when choosing appropriate graphical responses for evaluating goodness-of-fit.

Simulating animal space use from fitted integrated Step‐Selection Functions (iSSF)

“…First, simulated and observed paths can be visually compared (Figure 3). If the model has been specified in a way that describes the data-generating process reasonably well, the observed path should not stand out when compared to simulated paths (Fieberg et al, 2023). Similar to our case study, one can evaluate whether the observed and simulated paths exhibit similar behaviour near roads, rivers, or other prominent environmental features.…”

“…elevation could also be important), but we conclude that Model 3 is already a significant improvement over Model 1. Many realisations of this simulation could be used to formally measure the predictive power of each model (Potts et al, 2022) and to formally evaluate model goodness-of-fit (Fieberg et al, 2023).…”

“…Simulations from iSSFs have been used to investigate emergent patterns of space use from fitted iSSFs (Potts & Schlägel, 2020;Signer et al, 2017), to model connectivity between different animal populations or habitat patches (Aiello et al, 2023;Hofmann et al, 2023;Whittington et al, 2022), and to evaluate and validate fitted models (Fieberg et al, 2023;Potts et al, 2022;Sells et al, 2023). Although analytical approximations of various estimation targets exist for some situations (Potts & Börger, 2023;Potts & Schlägel, 2020), simulations are more intuitive, flexible, and applicable to a wider range of problems.…”

Predicting fine-scale distributions and emergent spatiotemporal patterns from temporally dynamic step selection simulations

“…Assessing the accuracy of simulations to different covariates is therefore important, and we suggest following a summary statistic approach as we have taken here, giving more weight to summary statistics that represent movement characteristics relating to the research question. An approach to assess the more qualitative aspects of the overall trajectory would be the 'lineup' approach (Fieberg et al, 2023), as many of the trajectories appeared visually similar to the observed data. We also advocate for using simulations as a means of model selection and to investigate drivers of animal movement decisions (Potts, Börger, et al, 2022).…”

Using lineups to evaluate goodness of fit of animal movement models

“…We have archived the data and code needed to reproduce all analyses in the paper in the Data Repository of the University of Minnesota (Fieberg et al., 2024), https://doi.org/10.13020/3m0d-mp29.…”