A practical guide to understanding and validating complex models using data simulations

Abstract

Biologists routinely fit novel and complex statistical models to push the limits of our understanding. Examples include, but are not limited to, flexible Bayesian approaches (e.g. BUGS, stan), frequentist and likelihood‐based approaches (e.g. packages lme4) and machine learning methods. These software and programs afford the user greater control and flexibility in tailoring complex hierarchical models. However, this level of control and flexibility places a higher degree of responsibility on the user to evaluate the robustness of their statistical inference. To determine how often biologists are running model diagnostics on hierarchical models, we reviewed 50 recently published papers in 2021 in the journal Nature Ecology & Evolution, and we found that the majority of published papers did not report any validation of their hierarchical models, making it difficult for the reader to assess the robustness of their inference. This lack of reporting likely stems from a lack of standardized guidance for best practices and standard methods. Here, we provide a guide to understanding and validating complex models using data simulations. To determine how often biologists use data simulation techniques, we also reviewed 50 recently published papers in 2021 in the journal Methods Ecology & Evolution. We found that 78% of the papers that proposed a new estimation technique, package or model used simulations or generated data in some capacity (18 of 23 papers); but very few of those papers (5 of 23 papers) included either a demonstration that the code could recover realistic estimates for a dataset with known parameters or a demonstration of the statistical properties of the approach. To distil the variety of simulations techniques and their uses, we provide a taxonomy of simulation studies based on the intended inference. We also encourage authors to include a basic validation study whenever novel statistical models are used, which in general, is easy to implement. Simulating data helps a researcher gain a deeper understanding of the models and their assumptions and establish the reliability of their estimation approaches. Wider adoption of data simulations by biologists can improve statistical inference, reliability and open science practices.

A Multifaceted Framework To Assess tradeoffs in Interpretability, Explanatory and Predictive Performances Of Alternative Joint Species Distribution Models

“…Similarly, adding traits or phylogeny can improve model interpretability. Future studies will be key to consolidate our findings on simulated case studies (Zurell, Berger, et al 2010; DiRenzo, Hanks, and Miller 2022), or across contrasted ecosystems, for instance dominated either by environmental filtering, or by competitive processes. Generalizing this approach across ecosystems will further help prioritize data collection effort on the long term.…”

“…Furthermore, how many non-target species should be included warrants further investigations. In the future, simulated datasets, overcoming limitations related to real world datasets (DiRenzo et al . 2022), could be used to more systematically assess how and to which extent jSDM performance is affected when increasing the number of non-target species and whether the functional or the trophic position of these species matters in this context.…”

“…To confirm the generality of our findings, future studies should focus on ecosystems that display different characteristics such as stronger environmental filtering or competitive processes; or by resorting to simulated case studies (Zurell et al . 2010 ; DiRenzo et al . 2022).…”

“…Furthermore, how many non-target species should be included warrants further investigations. In the future, simulated datasets, overcoming limitations related to real world datasets (DiRenzo et al 2022), could be used to more systematically assess how and to which extent jSDM performance is affected when increasing the number of non-target species and whether the functional or the trophic position of these species matters in this context. While species communities and assemblages are largely defined arbitrarily (Stroud et al 2015), such sensitivity analysis could help delineate which ecological units to include (or not) to improve model performance for the species of management/conservation interests.…”

“…While the list of additional species to consider can be prioritized based on existing knowledge in well-studied ecosystems, such information is often unavailable. Furthermore, a specific investigation, that might rely on simulated datasets to overcome limitations related to real world datasets (DiRenzo, Hanks, and Miller 2022), would be required to determine specific criteria, as well as optimal number of non-target species to include. While species communities and assemblages are largely defined arbitrarily (Stroud et al 2015), a systematic assessment of jSDM performance as increasing the number and types (for instance based on their functional or trophic roles) of non-target species would be valuable to optimise model performance for the species of management interests.…”