Evaluation framework for systems models

Abstract

As decisions in drug development increasingly rely on predictions from mechanistic systems models, assessing the predictive capability of such models is becoming more important. Several frameworks for the development of quantitative systems pharmacology (QSP) models have been proposed. In this paper, we add to this body of work with a framework that focuses on the appropriate use of qualitative and quantitative model evaluation methods. We provide details and references for those wishing to apply these methods, which include sensitivity and identifiability analyses, as well as concepts such as validation and uncertainty quantification. Many of these methods have been used successfully in other fields, but are not as common in QSP modeling. We illustrate how to apply these methods to evaluate QSP models, and propose methods to use in two case studies. We also share examples of misleading results when inappropriate analyses are used.

Current practices for QSP model assessment: an IQ consortium survey

“…A risk-based framework for verification and validation, as described previously [26], can be applied although the details need to be carefully considered given that few QSP projects are alike. Other such frameworks do exist in the literature [9,[28][29][30] , , and should be considered. While we are hesitant to prescribe a specific framework, modelers are encouraged to adopt one to better communicate model risk and uncertainty, enabling both technical and non-technical stakeholders to interpret simulation results appropriately.…”

“…ordinary differential equations, agent based, etc. ), understanding how these models are evaluated and assessed is of critical importance [9]. While the general process of fitting the model to a set of data (calibration) and checking against a dataset not used for fitting (qualification/validation) is typical [3], the details and metrics by which to conclude a model is suitable for its intended purpose can vary considerably, especially when considering factors such as data availability, data quality, model's intended use, knowledge of relevant biology, and model complexity.…”

Mechanistic Models of COVID-19: Insights into Disease Progression, Vaccines, and Therapeutics

“…With COVID-19, the dynamics of many serum immune biomarkers such as antibodies (IgM/IgG/IgA), cytokines (interferons, interleukins, chemokines…), and immune cells (B cells, T cells, monocytes…) have been measured with unprecedented resolution, including in deeper, immunologically-active sites such as draining lymph nodes, thus yielding opportunities for parameterization of large disease models. Careful calibration-qualification cycles with independent training and testing datasets can be performed to prevent model overparameterization, and robust QSP workflows [33][34][35] are available that may guide modelling effort s in this regard. Further, nonlinear mixed-effects approaches for parameter estimation can be employed to compare and rank multiple complex QSP models, each based on mutually exclusive scientific hypotheses, against multiple streams of pathogen-immune dynamics data.…”

Computational biology How mathematical modelling can help cure cancer

“…The first step in computational modelling is knowing what you want to model. This may seem obvious but being clear about this will determine your choice of model and analysis, as recently emphasised by several researchers in the field (eg, Fogarty et al, 2022, Braakman et al, 2022. This conceptual phase is then followed by building the actual model and writing the code for the simulation.…”

“…This conceptual phase is then followed by building the actual model and writing the code for the simulation. While doing so, one needs to consider how the model can be verified: this step consists of estimating the parameters and calibrating the model (Braakman et al, 2022). As Musuamba et al (2021) describe, verification can be seen as solving the equation in the right way, while the following step -validationconsists of solving the right equation.…”