Experimental Design and Data Analysis in Computer Simulation Studies in the Behavioral Sciences

Harwell, Michael R.; Kohli, Nidhi; Peralta, Yadira

doi:10.22237/jmasm/1509494520

Cited by 5 publications

(6 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For the epidemic outcomes in the absence of intervention, we performed the regressions for each outcome based on a set of parameter values generated following a full factorial design of experiments (DoE) (Table A1). [HKP17; MWC19]…”

Section: Simulations and Analysesmentioning

confidence: 99%

Effects of Mitigation and Control Policies in Realistic Epidemic Models Accounting for Household Transmission Dynamics

Alarid‐Escudero

Andrews

Goldhaber-Fiebert

2023

Preprint

View full text Add to dashboard Cite

Background: Compartmental infectious disease (ID) models are often used to evaluate non-pharmaceutical interventions (NPIs) and vaccines. Such models rarely separate within-household and community transmission, potentially introducing biases. We formulated an approach that incorporates household structure into ID models, extending the work of House and Keeling. Methods: We developed a multi-compartment susceptible-exposed-infectious-recovered-susceptible-vaccinated (MC-SEIRSV) modeling framework, allowing non-exponentially distributed duration in Exposed and Infectious compartments, that tracks within-household and community transmission. We simulated epidemics that varied systematically by community and household transmission rates, waning immunity rate, household size (3 or 5 members), and numbers of Exposed and Infectious compartments (1-3 each). We calibrated otherwise identical models without household structure to the early phase of each parameter combination's epidemic curve. We compared each model pair in terms of epidemic forecasts and predicted NPI and vaccine impacts on: the timing and magnitude of the epidemic peak and on its total size. Meta-analytic regressions characterized the relationship between household structure inclusion and the size and direction of biases. Results: Otherwise similar models with and without household structure produced equivalent early epidemic curves. However, forecasts from models without household structure were biased. Without intervention, they were upward-biased on peak size and total epidemic size, with biases also depending on the number of Exposed and Infectious compartments. Model-estimated NPI effects of a 60% reduction in community contacts on peak time and size were systematically overestimated without household structure. When the NPI yielded a 20% reduction, biases were smaller. Because vaccination impacted both community and household transmission, their biases were still smaller. Conclusions: ID models without household structure can produce biased outcomes. Analysts should consider incorporating household structure.

show abstract

Section: Simulations and Analysesmentioning

confidence: 99%

Effects of Mitigation and Control Policies in Realistic Epidemic Models Accounting for Household Transmission Dynamics

Alarid‐Escudero

Andrews

Goldhaber-Fiebert

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…For epidemic outcomes in the absence of intervention, we estimate the regressions for each outcome based on a set of parameter values generated following a full factorial design of experiments (DoE; Appendix Table A1). 14,15 We use the same general approach (i.e., DoE design and meta-regression) for intervention effects. We focus on the intervention effects of NPIs that would be differentially delivered in communities versus households (e.g., business closures) without vaccination.…”

Section: Design Of Experimentsmentioning

confidence: 99%

Effects of Mitigation and Control Policies in Realistic Epidemic Models Accounting for Household Transmission Dynamics

Alarid-Escudero,

Andrews,

Goldhaber-Fiebert

2023

Med Decis Making

View full text Add to dashboard Cite

Background Compartmental infectious disease (ID) models are often used to evaluate nonpharmaceutical interventions (NPIs) and vaccines. Such models rarely separate within-household and community transmission, potentially introducing biases in situations in which multiple transmission routes exist. We formulated an approach that incorporates household structure into ID models, extending the work of House and Keeling. Design We developed a multicompartment susceptible-exposed-infectious-recovered-susceptible-vaccinated (MC-SEIRSV) modeling framework, allowing nonexponentially distributed duration in exposed and infectious compartments, that tracks within-household and community transmission. We simulated epidemics that varied by community and household transmission rates, waning immunity rate, household size (3 or 5 members), and numbers of exposed and infectious compartments (1–3 each). We calibrated otherwise identical models without household structure to the early phase of each parameter combination’s epidemic curve. We compared each model pair in terms of epidemic forecasts and predicted NPI and vaccine impacts on the timing and magnitude of the epidemic peak and its total size. Meta-analytic regressions characterized the relationship between household structure inclusion and the size and direction of biases. Results Otherwise similar models with and without household structure produced equivalent early epidemic curves. However, forecasts from models without household structure were biased. Without intervention, they were upward biased on peak size and total epidemic size, with biases also depending on the number of exposed and infectious compartments. Model-estimated NPI effects of a 60% reduction in community contacts on peak time and size were systematically overestimated without household structure. Biases were smaller with a 20% reduction NPI. Because vaccination affected both community and household transmission, their biases were smaller. Conclusions ID models without household structure can produce biased outcomes in settings in which within-household and community transmission differ. Highlights Infectious disease models rarely separate household transmission from community transmission. The pace of household transmission may differ from community transmission, depends on household size, and can accelerate epidemic growth. Many infectious disease models assume exponential duration distributions for infected states. However, the duration of most infections is not exponentially distributed, and distributional choice alters modeled epidemic dynamics and intervention effectiveness. We propose a mathematical framework for household and community transmission that allows for nonexponential duration times and a suite of interventions and quantified the effect of accounting for household transmission by varying household size and duration distributions of infected states on modeled epidemic dynamics. Failure to include household structure induces biases in the modeled overall course of an epidemic and the effects of interventions delivered differentially in community settings. Epidemic dynamics are faster and more intense in populations with larger household sizes and for diseases with nonexponentially distributed infectious durations. Modelers should consider explicitly incorporating household structure to quantify the effects of non-pharmaceutical interventions (e.g., shelter-in-place).

show abstract

“…Several articles have used DAE tools to report their research results, for example, Culpepper & Aguinis (2011). Others, such as Skrondal (2000), Harwell, Kohli & Peralta (2017), and Morris, White & Crowther (2019), provide general advice in the planning and analysis of simulation studies. Both Skrondal (2000) and Harwell, Kohli & Peralta (2017) argue for the use of a factorial design approach for planning and a "meta-model" for analysis.…”

Section: Introductionmentioning

confidence: 99%

“…Others, such as Skrondal (2000), Harwell, Kohli & Peralta (2017), and Morris, White & Crowther (2019), provide general advice in the planning and analysis of simulation studies. Both Skrondal (2000) and Harwell, Kohli & Peralta (2017) argue for the use of a factorial design approach for planning and a "meta-model" for analysis. Skrondal (2000) is particularly similar to our approach, although generalized linear models (GLMs) are favoured over the simpler ANOVA we demonstrate.…”

Section: Introductionmentioning

confidence: 99%

Let's practice what we preach: Planning and interpreting simulation studies with design and analysis of experiments

Chipman

Bingham

2022

Can J Statistics

View full text Add to dashboard Cite

Statisticians recommend design and analysis of experiments (DAE) for evidence-based research but often use tables to present their own simulation studies. Could DAE do better? We outline how DAE methods can be used to plan and analyze simulation studies. Tools for planning include cause-and-effect diagrams and factorial and fractional factorial designs. Analysis is carried out via analysis of variance, main effect and interaction plots, and other DAE tools. We also demonstrate how Taguchi robust parameter design can be used to study the robustness of methods to a variety of uncontrollable population parameters.Résumé: Les statisticiens prônent le recours aux plans et analyse d'expériences (DAE) en recherche factuelle, mais le plus souvent, ils se contentent de tableaux pour présenter leurs propres études de simulation. Les auteurs de ce travail examinent l'apport et l'amélioration que pourrait apporter l'adoption de l'approche (DAE). Pour y répondre, ils présentent une description de la façon dont les méthodes DAE peuvent être employées pour planifier et analyser les études de simulation. Les outils de planification proposés comprennent les diagrammes de cause à effet et les plans factoriels et fractionnaires. L'analyse préconisée repose sur l'ANOVA, les graphiques d'effets principaux et d'interactions et d'autres outils DAE. Les auteurs montrent également comment les plans robustes de Taguchi peuvent être utilisés pour étudier la robustesse des méthodes à divers paramètres de population non contrôlables.

show abstract

Experimental Design and Data Analysis in Computer Simulation Studies in the Behavioral Sciences

Cited by 5 publications

References 42 publications

Effects of Mitigation and Control Policies in Realistic Epidemic Models Accounting for Household Transmission Dynamics

Effects of Mitigation and Control Policies in Realistic Epidemic Models Accounting for Household Transmission Dynamics

Effects of Mitigation and Control Policies in Realistic Epidemic Models Accounting for Household Transmission Dynamics

Let's practice what we preach: Planning and interpreting simulation studies with design and analysis of experiments

Contact Info

Product

Resources

About