Multiobjective Calibration of Disease Simulation Models Using Gaussian Processes

Sai, Aditya; Vivas-Valencia, Carolina; Imperiale, Thomas F.; Kong, Nan

doi:10.1177/0272989x19862560

Cited by 7 publications

(8 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…8 and our own literature search on PubMed to capture articles published between 2018 and January 2020 and/or studies that were missed in the review. We found 5 studies 9,21–24 of 300 PubMed search results (search took place in March 2020; see Supplementary Figure S2) in addition to the 13 studies found in Degeling et al 8 The most common choices in health economics are linear regression (LM) and Gaussian GP, with a squared exponential covariance matrix. 8 Alternative choices include GAMs, 16,25 ANNs, 12,14 GP using Matern and rational quadratic covariance matrix, 22 and symbolic regression.…”

Section: Methodsmentioning

confidence: 93%

“…We found 5 studies 9,21-24 of 300 PubMed search results (search took place in March 2020; see Supplementary Figure S2) in addition to the 13 studies found in Degeling et al 8 The most common choices in health economics are linear regression (LM) and Gaussian GP, with a squared exponential covariance matrix. 8 Alternative choices include GAMs, 16,25 ANNs, 12,14 GP using Matern and rational quadratic covariance matrix, 22 and symbolic regression. 26 Although symbolic regression is valuable because it does not assume any prior model structure, it is relatively more difficult to implement and therefore was excluded from this study (Table 1).…”

Section: Metamodeling Steps For Uncertainty Quantificationmentioning

confidence: 99%

See 1 more Smart Citation

Choosing a Metamodel of a Simulation Model for Uncertainty Quantification

et al. 2021

View full text Add to dashboard Cite

Background Metamodeling may substantially reduce the computational expense of individual-level state transition simulation models (IL-STM) for calibration, uncertainty quantification, and health policy evaluation. However, because of the lack of guidance and readily available computer code, metamodels are still not widely used in health economics and public health. In this study, we provide guidance on how to choose a metamodel for uncertainty quantification. Methods We built a simulation study to evaluate the prediction accuracy and computational expense of metamodels for uncertainty quantification using life-years gained (LYG) by treatment as the IL-STM outcome. We analyzed how metamodel accuracy changes with the characteristics of the simulation model using a linear model (LM), Gaussian process regression (GP), generalized additive models (GAMs), and artificial neural networks (ANNs). Finally, we tested these metamodels in a case study consisting of a probabilistic analysis of a lung cancer IL-STM. Results In a scenario with low uncertainty in model parameters (i.e., small confidence interval), sufficient numbers of simulated life histories, and simulation model runs, commonly used metamodels (LM, ANNs, GAMs, and GP) have similar, good accuracy, with errors smaller than 1% for predicting LYG. With a higher level of uncertainty in model parameters, the prediction accuracy of GP and ANN is superior to LM. In the case study, we found that in the worst case, the best metamodel had an error of about 2.1%. Conclusion To obtain good prediction accuracy, in an efficient way, we recommend starting with LM, and if the resulting accuracy is insufficient, we recommend trying ANNs and eventually also GP regression.

show abstract

Section: Methodsmentioning

confidence: 93%

Section: Metamodeling Steps For Uncertainty Quantificationmentioning

confidence: 99%

Choosing a Metamodel of a Simulation Model for Uncertainty Quantification

et al. 2021

View full text Add to dashboard Cite

show abstract

“…com/mclements/prostata. Prakash et al, 46 Sai et al 49 CMOST is a microsimulation model for modeling the natural history of colorectal cancer, simulating the effects of colorectal cancer screening interventions, and calculating the resulting costs. According to the authors, several computational microsimulation tools have been reported for estimating efficiency and costeffectiveness of colorectal cancer prevention but none of these tools is publicly available.…”

Section: Authorsmentioning

confidence: 99%

“…Although this situation has changed since 2007, the related work on open-source frameworks for this domain is still limited. For example, a Scopus search in May 2020 with the TITLE-ABS-KEY search terms (open AND source AND framework AND microsimulation AND health) yielded only two results.46;26 Similarly, a PubMed search with the same keywords also returned only two studies.46;49 As documented in Table5, Prakash et al46 developed and used a specific open-source tool for colorectal cancer microsimulation that was later used by Sai et al,49 while Kuchenbecker et al…”

mentioning

confidence: 99%

Simulation Framework for Realistic Large-scale Individual-level Data Generation with an Application in the Health Domain

Tikka¹,

Hakanen²,

Saarela³

et al. 2020

Preprint

View full text Add to dashboard Cite

We propose a general framework for realistic data generation and simulation of complex systems in the health domain. The main use cases of the framework are predicting the development of risk factors and disease occurrence, evaluating the impact of interventions and policy decisions, and statistical method development. We present the fundamentals of the framework using rigorous mathematical definitions. The framework supports calibration to a real population as well as various manipulations and data collection processes. The freely available open-source implementation in R embraces efficient data structures, parallel computing and fast random number generation which ensure reproducibility and scalability. With the framework it is possible to run daily-level simulations for populations of millions individuals for decades of simulated time. An example on the occurrence of stroke, type 2 diabetes and mortality illustrates the usage of the framework in the Finnish context. In the example, we demonstrate the data-collection functionality by studying the impact of non-participation on the estimated risk models.

show abstract

“…Sai et al [ 24 ] investigated the efficiency of a Gaussian Processes-based surrogate modeling approach to approximate the CMOST model to alleviate the computational burden in calibrating the CMOST model. Compared to above papers in the literature, we studied a different version of the calibration problem, for which we have the option of using a baseline parameter design from the literature and/or previous studies to start the model parameter adjustments.…”

Section: Introductionmentioning

confidence: 99%

A two-phase approach to re-calibrating expensive computer simulation for sex-specific colorectal neoplasia development modeling

Vivas-Valencia

Zhou

Sai

et al. 2022

BMC Med Inform Decis Mak

Self Cite

View full text Add to dashboard Cite

Background Medical evidence from more recent observational studies may significantly alter our understanding of disease incidence and progression, and would require recalibration of existing computational and predictive disease models. However, it is often challenging to perform recalibration when there are a large number of model parameters to be estimated. Moreover, comparing the fitting performances of candidate parameter designs can be difficult due to significant variation in simulated outcomes under limited computational budget and long runtime, even for one simulation replication. Methods We developed a two-phase recalibration procedure. As a proof-of-the-concept study, we verified the procedure in the context of sex-specific colorectal neoplasia development. We considered two individual-based state-transition stochastic simulation models, estimating model parameters that govern colorectal adenoma occurrence and its growth through three preclinical states: non-advanced precancerous polyp, advanced precancerous polyp, and cancerous polyp. For the calibration, we used a weighted-sum-squared error between three prevalence values reported in the literature and the corresponding simulation outcomes. In phase 1 of the calibration procedure, we first extracted the baseline parameter design from relevant studies on the same model. We then performed sampling-based searches within a proper range around the baseline design to identify the initial set of good candidate designs. In phase 2, we performed local search (e.g., the Nelder-Mead algorithm), starting from the candidate designs identified at the end of phase 1. Further, we investigated the efficiency of exploring dimensions of the parameter space sequentially based on our prior knowledge of the system dynamics. Results The efficiency of our two-phase re-calibration procedure was first investigated with CMOST, a relatively inexpensive computational model. It was then further verified with the V/NCS model, which is much more expensive. Overall, our two-phase procedure showed a better goodness-of-fit than the straightforward employment of the Nelder-Mead algorithm, when only a limited number of simulation replications were allowed. In addition, in phase 2, performing local search along parameter space dimensions sequentially was more efficient than performing the search over all dimensions concurrently. Conclusion The proposed two-phase re-calibration procedure is efficient at estimating parameters of computationally expensive stochastic dynamic disease models. Supplementary Information The online version contains supplementary material available at 10.1186/s12911-022-01991-7.

show abstract

Multiobjective Calibration of Disease Simulation Models Using Gaussian Processes

Cited by 7 publications

References 28 publications

Choosing a Metamodel of a Simulation Model for Uncertainty Quantification

Choosing a Metamodel of a Simulation Model for Uncertainty Quantification

Simulation Framework for Realistic Large-scale Individual-level Data Generation with an Application in the Health Domain

A two-phase approach to re-calibrating expensive computer simulation for sex-specific colorectal neoplasia development modeling

Contact Info

Product

Resources

About