Adapting modeling and simulation credibility standards to computational systems biology

Abstract

Computational models are increasingly used in high-impact decision making in science, engineering, and medicine. The National Aeronautics and Space Administration (NASA) uses computational models to perform complex experiments that are otherwise prohibitively expensive or require a microgravity environment. Similarly, the Food and Drug Administration (FDA) and European Medicines Agency (EMA) have began accepting models and simulations as forms of evidence for pharmaceutical and medical device approval. It is crucial that computational models meet a standard of credibility when using them in high-stakes decision making. For this reason, institutes including NASA, the FDA, and the EMA have developed standards to promote and assess the credibility of computational models and simulations. However, due to the breadth of models these institutes assess, these credibility standards are mostly qualitative and avoid making specific recommendations. On the other hand, modeling and simulation in systems biology is a narrower domain and several standards are already in place. As systems biology models increase in complexity and influence, the development of a credibility assessment system is crucial. Here we review existing standards in systems biology, credibility standards in other science, engineering, and medical fields, and propose the development of a credibility standard for systems biology models.

Veterinary systems biology for bridging the phenotype–genotype gap via computational modeling for disease epidemiology and animal welfare

“…It is important to ensure that the model behaves like biological systems [ 98 ]. Several steps are involved in the model validation process, including literature mining, comparing the model output with experimental data, sensitivity analysis, optimizing the model through parameter estimation, conducting robustness testing and refining the model if errors are observed by adjusting parameters, reaction kinetics, to ensure accurate predictions [ 99–101 ]. A previous study established a network-based model for simulating the transmission of porcine reproductive and respiratory syndrome virus (PRRSV) between farms.…”

A framework for multi-scale intervention modeling: virtual cohorts, virtual clinical trials, and model-to-model comparisons

“…We validated our virtual hosts at multiple scales according to the ten simple rules credibility standard (Erdemir et al, 2020;Fogarty et al, 2022;Nanda et al, 2023;Tatka et al, 2023). Figures 3A-E shows trajectories of 6,500 primary granulomas and whole-host CFU counts taken from 500 virtual hosts generated after we calibrated to multiple datasets from different NHP studies.…”

“…In order for a model to credibly perform credible in silico experiments requires rigorous validation against available data (Tatka et al, 2023). The precision and rigor required are systemspecific and adapted to the expected use of the model's output (Fogarty et al, 2022), and consequences of incorrect model predictions (Aldieri et al, 2023).…”

“…The precision and rigor required are systemspecific and adapted to the expected use of the model's output (Fogarty et al, 2022), and consequences of incorrect model predictions (Aldieri et al, 2023). Various standards exist to codify model validation (Fogarty et al, 2022;Tatka et al, 2023); including the ten rules for model credibility developed by the Multiscale Modeling Consortium (Erdemir et al, 2020;Fogarty et al, 2022;Nanda et al, 2023;Tatka et al, 2023) for systems biology approaches, as well as the ASME VandV40 standards (ASME, 2018;Aldieri et al, 2023;Tatka et al, 2023), and NASA standards for models and simulation (NASA, 2016;Tatka et al, 2023). Each of these standards establishes a series of assessments by which we can establish the appropriateness of a model to address a given question of interest relative to the model's context of use.…”

“…With our ability to select amongst many types of models that can credibly represent the same system, we need a methodology to compare models in an implementation-agnostic way. We have seen a recent push to standardize modeling approaches with modeling ecosystems such as CompuCell3D (Poplawski et al, 2008;Shirinifard et al, 2009), VCell (Blinov et al, 2008;Schaff et al, 2016), PhysiCell (Ghaffarizadeh et al, 2018), as well as standardized language for ODE model implementation such as SBML (Keating et al, 2020), SED-ML (Bergmann et al, 2017;Smith et al, 2021), COMBINE, OMEX (Bergmann et al, 2014;Neal et al, 2020), and others (Tatka et al, 2023). With this variety of platforms, software, computational frameworks, and databases available (computational models, medical digital twins, etc.…”