Ecosystem Model Skill Assessment. Yes We Can!

Abstract

Need to Assess the Skill of Ecosystem ModelsAccelerated changes to global ecosystems call for holistic and integrated analyses of past, present and future states under various pressures to adequately understand current and projected future system states. Ecosystem models can inform management of human activities in a complex and changing environment, but are these models reliable? Ensuring that models are reliable for addressing management questions requires evaluating their skill in representing real-world processes and dynamics. Skill has been evaluated for just a limited set of some biophysical models. A range of skill assessment methods have been reviewed but skill assessment of full marine ecosystem models has not yet been attempted.Northeast US Atlantis Marine Ecosystem ModelWe assessed the skill of the Northeast U.S. (NEUS) Atlantis marine ecosystem model by comparing 10-year model forecasts with observed data. Model forecast performance was compared to that obtained from a 40-year hindcast. Multiple metrics (average absolute error, root mean squared error, modeling efficiency, and Spearman rank correlation), and a suite of time-series (species biomass, fisheries landings, and ecosystem indicators) were used to adequately measure model skill. Overall, the NEUS model performed above average and thus better than expected for the key species that had been the focus of the model tuning. Model forecast skill was comparable to the hindcast skill, showing that model performance does not degenerate in a 10-year forecast mode, an important characteristic for an end-to-end ecosystem model to be useful for strategic management purposes.Skill Assessment Is Both Possible and AdvisableWe identify best-practice approaches for end-to-end ecosystem model skill assessment that would improve both operational use of other ecosystem models and future model development. We show that it is possible to not only assess the skill of a complicated marine ecosystem model, but that it is necessary do so to instill confidence in model results and encourage their use for strategic management. Our methods are applicable to any type of predictive model, and should be considered for use in fields outside ecology (e.g. economics, climate change, and risk assessment).

Spatio‐temporal models of intermediate complexity for ecosystem assessments: A new tool for spatial fisheries management

“…Livingston et al, 2017). We therefore support ongoing comparative research using multiple ecological models when evaluating climate or human-mediated changes on marine ecosystems (Kaplan et al, 2018;Olsen et al, 2016;Pope et al, 2019;Spence et al, 2018;Tittensor et al, 2018), and note that the MICE-in-space model could fill a useful niche in these model portfolios. Finally, nonconsumptive processes may cause diet analyses to misrepresent the cumulative impact of changing biomass for one species on per-capita productivity for other species.…”

“…Comparing results from multiple ecosystem models can help to evaluate the sensitivity of estimated ecosystem properties to structural assumptions and multiple data sources. We therefore support ongoing comparative research using multiple ecological models when evaluating climate or human-mediated changes on marine ecosystems (Kaplan et al, 2018;Olsen et al, 2016;Pope et al, 2019;Spence et al, 2018;Tittensor et al, 2018), and note that the MICE-in-space model could fill a useful niche in these model portfolios.…”

Climate change impacts on southern Ross Sea phytoplankton composition, productivity, and export

“…Model-data fit of the contemporary control run was assessed quantitatively using statistical skill metrics that accentuate specific aspects of model performance [Stow et al, 2009;Olsen et al, 2016], including root mean squared difference (RMSD), unbiased RMSD, bias, standard deviation, and linear correlation. Bias, unbiased RMSD, and RMSD were plotted using Target diagrams [Hofmann et al, 2008;Jolliff et al, 2009;Friedrichs et al, 2009].…”

A Framework for Incorporating Species, Fleet, Habitat, and Climate Interactions into Fishery Management

“…Full ecosystem models can also be useful in addressing species interactions (as well as climate, habitat, and fleet interactions; see below). A spatially explicit bio-geochemical end-to-end ecosystem model, Atlantis (Fulton et al, 2011), has been developed for the full Northeast US shelf , and compared with recent data on multispecies and ecosystem trends (Olsen et al, 2016).…”