How Many Costly Simulations Do we Need to Create Accurate Metamodels? A Case Study on Predicting HIV Viral Load in Response to Clinically Relevant Intervention Scenarios

Lutz, Christopher B; Giabbanelli, Philippe J.; Fisher, Andrew; Mago, Vijay

doi:10.23919/annsim52504.2021.9552036

Cited by 2 publications

(1 citation statement)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…19 Other studies on metamodeling have shown that the results from limited availability of data are also model-sensitive, as two different metamodels were shown to perform as well as the respective models trained on the full data set with just 5% and 25% of the data set, respectively. 20 Since the structure of a model can be an important confounding factor, we consider network simulations for different phenomena (physical viral spread, online rumor spread, power networks) and investigate verification performances on several models (i.e., three types of viral spread and three types of rumor spread). This complements our past research in which performances used a single model type and data limitation was not a concern.…”

Section: Introductionmentioning

confidence: 99%

Experimental evaluation of a machine learning approach to improve the reproducibility of network simulations

Liang,

Phan,

Giabbanelli

2024

SIMULATION

View full text Add to dashboard Cite

A stochastic network simulation is verified when its distribution of outputs is aligned with the ground truth, while tolerating deviations due to variability in real-world measurements and the randomness of a stochastic simulation. However, comparing distributions may yield false positives, as erroneous simulations may have the expected distribution yet present aberrations in low-level patterns. For instance, the number of sick individuals may present the right trend over time, but the wrong individuals were infected. We previously proposed an approach that transforms simulation traces into images verified by machine learning algorithms that account for low-level patterns. We demonstrated the viability of this approach when many simulation traces are compared with a large ground truth data set. However, ground truth data are often limited. For example, a publication may include few images of their simulation as illustrations; hence, teams that independently re-implement the model can only compare low-level patterns with few cases. In this paper, we examine whether our approach can be utilized with very small data sets (e.g., 5–10 images), as provided in publications. Depending on the network simulation model (e.g., rumor spread, cascading failure, and disease spread), we show that results obtained with little data can even surpass results obtained with moderate amounts of data at the cost of variability. Although a good accuracy is obtained in detecting several forms of errors, this paper is only a first step in the use of this technique for verification; hence, future works should assess the applicability of our approach to other types of network simulations.

show abstract

Section: Introductionmentioning

confidence: 99%

Experimental evaluation of a machine learning approach to improve the reproducibility of network simulations

Liang,

Phan,

Giabbanelli

2024

SIMULATION

View full text Add to dashboard Cite

show abstract

When do we need massive computations to perform detailed COVID-19 simulations?

Lutz

Giabbanelli

2021

Preprint

View full text Add to dashboard Cite

The COVID-19 pandemic has infected over 200 million people worldwide and killed more than 4 million as of August 2021. Many intervention strategies have been utilized by governments around the world, including masks, social distancing, and vaccinations. However, officials making decisions regarding interventions may have a limited time to act. Computer simulations can aid them by predicting future disease outcomes, but they also have limitations due to requirements on processing power or time. This paper examines whether a machine learning model can be trained on a small subset of simulation runs to inexpensively predict future disease trajectories very close to the original simulation results. Using four previously published agent-based models for COVID-19, this paper analyzes the predictions of decision tree regression machine learning models and compares them to the results of the original simulations. The results indicate that accurate machine learning meta-models can be generated from simulation models with no strong interventions (e.g., vaccines, lockdowns) using small amounts of simulation data. However, meta-models for simulation models that include strong interventions required much more training data to achieve a similar accuracy. This indicates that machine learning meta-models could be used in some scenarios to assist in faster decision making.

show abstract

How Many Costly Simulations Do we Need to Create Accurate Metamodels? A Case Study on Predicting HIV Viral Load in Response to Clinically Relevant Intervention Scenarios

Cited by 2 publications

References 32 publications

Experimental evaluation of a machine learning approach to improve the reproducibility of network simulations

Experimental evaluation of a machine learning approach to improve the reproducibility of network simulations

When do we need massive computations to perform detailed COVID-19 simulations?

Contact Info

Product

Resources

About