Choosing a Metamodel of a Simulation Model for Uncertainty Quantification

Abstract

Background Metamodeling may substantially reduce the computational expense of individual-level state transition simulation models (IL-STM) for calibration, uncertainty quantification, and health policy evaluation. However, because of the lack of guidance and readily available computer code, metamodels are still not widely used in health economics and public health. In this study, we provide guidance on how to choose a metamodel for uncertainty quantification. Methods We built a simulation study to evaluate the prediction accuracy and computational expense of metamodels for uncertainty quantification using life-years gained (LYG) by treatment as the IL-STM outcome. We analyzed how metamodel accuracy changes with the characteristics of the simulation model using a linear model (LM), Gaussian process regression (GP), generalized additive models (GAMs), and artificial neural networks (ANNs). Finally, we tested these metamodels in a case study consisting of a probabilistic analysis of a lung cancer IL-STM. Results In a scenario with low uncertainty in model parameters (i.e., small confidence interval), sufficient numbers of simulated life histories, and simulation model runs, commonly used metamodels (LM, ANNs, GAMs, and GP) have similar, good accuracy, with errors smaller than 1% for predicting LYG. With a higher level of uncertainty in model parameters, the prediction accuracy of GP and ANN is superior to LM. In the case study, we found that in the worst case, the best metamodel had an error of about 2.1%. Conclusion To obtain good prediction accuracy, in an efficient way, we recommend starting with LM, and if the resulting accuracy is insufficient, we recommend trying ANNs and eventually also GP regression.

When Is Mass Prophylaxis Cost-Effective for Epidemic Control? A Comparison of Decision Approaches

“…This is consistent with results achieved in previous meta-modeling studies. 11,18,21 Given the high accuracy of the meta-models we created, it is likely that similarly high accuracy could be achieved with a smaller training set. 42 This could make meta-modeling more feasible in low-resource settings.…”

“…6 The resulting healthrelated quality of life of a noninfected individual is 0.83. 6 With the base-case parameter values, we can write the decision rule (18) as…”

“…Common methods for developing meta-models include linear and logistic regression, generalized linear and additive models, random forest and tree methods, and neural networks. 12,17,18 The typical tradeoff in meta-modeling is between reducing computational burden versus potentially introducing new sources of error relative to the real-world system and losing interpretability with advanced machine learning methods. 17,19 Logistic and linear regression models are relatively interpretable.…”

“…20 Metamodels have proven useful in a variety of health economics applications because of their reduced computational requirements, especially in model calibration, uncertainty quantification, policy evaluation, development of tools for real-time decision making, and rapid reanalysis of existing work. 10,11,18,19,21,22 In this article, we aim to answer the following question: ''How can we quickly and accurately estimate the cost-effectiveness of mass prophylaxis for epidemic control?'' We compare 2 different approaches.…”

“…18,21 However, linear and logistic regression models rely on the strong assumption of linearity, so alternative models may be more accurate. 18 Advanced machine learning models, such as neural networks and random forests, may better approximate an original model but are less interpretable. 20 The level of interpretability represents a key barrier in the willingness to adopt machine learning models in medical and public health settings, as it can be difficult to audit the decisions of these models.…”