Health care delivery systems are inherently complex, consisting of multiple tiers of interdependent subsystems and processes that are adaptive to changes in the environment and behave in a nonlinear fashion. Traditional health technology assessment and modeling methods often neglect the wider health system impacts that can be critical for achieving desired health system goals and are often of limited usefulness when applied to complex health systems. Researchers and health care decision makers can either underestimate or fail to consider the interactions among the people, processes, technology, and facility designs. Health care delivery system interventions need to incorporate the dynamics and complexities of the health care system context in which the intervention is delivered. This report provides an overview of common dynamic simulation modeling methods and examples of health care system interventions in which such methods could be useful. Three dynamic simulation modeling methods are presented to evaluate system interventions for health care delivery: system dynamics, discrete event simulation, and agent-based modeling. In contrast to conventional evaluations, a dynamic systems approach incorporates the complexity of the system and anticipates the upstream and downstream consequences of changes in complex health care delivery systems. This report assists researchers and decision makers in deciding whether these simulation methods are appropriate to address specific health system problems through an eight-point checklist referred to as the SIMULATE (System, Interactions, Multilevel, Understanding, Loops, Agents, Time, Emergence) tool. It is a primer for researchers and decision makers working in health care delivery and implementation sciences who face complex challenges in delivering effective and efficient care that can be addressed with system interventions. On reviewing this report, the readers should be able to identify whether these simulation modeling methods are appropriate to answer the problem they are addressing and to recognize the differences of these methods from other modeling approaches used typically in health technology assessment applications.

BackgroundAcute Kidney Injury (AKI) occurs in at least 5 % of hospitalized patients and can result in 40–70 % morbidity and mortality. Even following recovery, many subjects may experience progressive deterioration of renal function. The heterogeneous etiology and pathophysiology of AKI complicates its diagnosis and medical management and can add to poor patient outcomes and incur substantial hospital costs. AKI is predictable and may be avoidable if early risk factors are identified and utilized in the clinical setting. Timely detection of undiagnosed AKI in hospitalized patients can also lead to better disease management.MethodsData from 25,521 hospital stays in one calendar year of patients 60 years and older was collected from a large health care system. Four machine learning models (logistic regression, support vector machines, decision trees and naïve Bayes) along with their ensemble were tested for AKI prediction and detection tasks. Patient demographics, laboratory tests, medications and comorbid conditions were used as the predictor variables. The models were compared using the area under ROC curve (AUC) evaluation metric.ResultsLogistic regression performed the best for AKI detection (AUC 0.743) and was a close second to the ensemble for AKI prediction (AUC ensemble: 0.664, AUC logistic regression: 0.660). History of prior AKI, use of combination drugs such as ACE inhibitors, NSAIDS and diuretics, and presence of comorbid conditions such as respiratory failure were found significant for both AKI detection and risk prediction.ConclusionsThe machine learning models performed fairly well on both predicting AKI and detecting undiagnosed AKI. To the best of our knowledge, this is the first study examining the difference between prediction and detection of AKI. The distinction has clinical relevance, and can help providers either identify at risk subjects and implement preventative strategies or manage their treatment depending on whether AKI is predicted or detected.

In a previous report, the ISPOR Task Force on Dynamic Simulation Modeling Applications in Health Care Delivery Research Emerging Good Practices introduced the fundamentals of dynamic simulation modeling and identified the types of health care delivery problems for which dynamic simulation modeling can be used more effectively than other modeling methods. The hierarchical relationship between the health care delivery system, providers, patients, and other stakeholders exhibits a level of complexity that ought to be captured using dynamic simulation modeling methods. As a tool to help researchers decide whether dynamic simulation modeling is an appropriate method for modeling the effects of an intervention on a health care system, we presented the System, Interactions, Multilevel, Understanding, Loops, Agents, Time, Emergence (SIMULATE) checklist consisting of eight elements. This report builds on the previous work, systematically comparing each of the three most commonly used dynamic simulation modeling methods-system dynamics, discrete-event simulation, and agent-based modeling. We review criteria for selecting the most suitable method depending on 1) the purpose-type of problem and research questions being investigated, 2) the object-scope of the model, and 3) the method to model the object to achieve the purpose. Finally, we provide guidance for emerging good practices for dynamic simulation modeling in the health sector, covering all aspects, from the engagement of decision makers in the model design through model maintenance and upkeep. We conclude by providing some recommendations about the application of these methods to add value to informed decision making, with an emphasis on stakeholder engagement, starting with the problem definition. Finally, we identify areas in which further methodological development will likely occur given the growing "volume, velocity and variety" and availability of "big data" to provide empirical evidence and techniques such as machine learning for parameter estimation in dynamic simulation models. Upon reviewing this report in addition to using the SIMULATE checklist, the readers should be able to identify whether dynamic simulation modeling methods are appropriate to address the problem at hand and to recognize the differences of these methods from those of other, more traditional modeling approaches such as Markov models and decision trees. This report provides an overview of these modeling methods and examples of health care system problems in which such methods have been useful. The primary aim of the report was to aid decisions as to whether these simulation methods are appropriate to address specific health systems problems. The report directs readers to other resources for further education on these individual modeling methods for system interventions in the emerging field of health care delivery science and implementation.

Objective To estimate population health outcomes with delayed second dose versus standard schedule of SARS-CoV-2 mRNA vaccination. Design Simulation agent based modeling study. Setting Simulated population based on real world US county. Participants The simulation included 100 000 agents, with a representative distribution of demographics and occupations. Networks of contacts were established to simulate potentially infectious interactions though occupation, household, and random interactions. Interventions Simulation of standard covid-19 vaccination versus delayed second dose vaccination prioritizing the first dose. The simulation runs were replicated 10 times. Sensitivity analyses included first dose vaccine efficacy of 50%, 60%, 70%, 80%, and 90% after day 12 post-vaccination; vaccination rate of 0.1%, 0.3%, and 1% of population per day; assuming the vaccine prevents only symptoms but not asymptomatic spread (that is, non-sterilizing vaccine); and an alternative vaccination strategy that implements delayed second dose for people under 65 years of age, but not until all those above this age have been vaccinated. Main outcome measures Cumulative covid-19 mortality, cumulative SARS-CoV-2 infections, and cumulative hospital admissions due to covid-19 over 180 days. Results Over all simulation replications, the median cumulative mortality per 100 000 for standard dosing versus delayed second dose was 226 v 179, 233 v 207, and 235 v 236 for 90%, 80%, and 70% first dose efficacy, respectively. The delayed second dose strategy was optimal for vaccine efficacies at or above 80% and vaccination rates at or below 0.3% of the population per day, under both sterilizing and non-sterilizing vaccine assumptions, resulting in absolute cumulative mortality reductions between 26 and 47 per 100 000. The delayed second dose strategy for people under 65 performed consistently well under all vaccination rates tested. Conclusions A delayed second dose vaccination strategy, at least for people aged under 65, could result in reduced cumulative mortality under certain conditions.

Introduction We report the first prospective analysis of human factors elements contributing to invasive procedural never events using a validated Human Factors Analysis and Classification System (HFACS). Methods From 8/2009 - 8/2014 surgical and invasive procedural “Never Events” (retained foreign object, wrong site/side procedure, wrong implant, wrong procedure) underwent systematic causation analysis promptly after the event. Contributing human factors were categorized using Reason's 4 levels of error causation and 161 HFACS subcategories (nano-codes). Results During the study approximately 1.5 million procedures were performed and 69 never events were identified. A total of 628 contributing human factors nano-codes were identified. Action-based errors (n=260) and preconditions to actions (n=296) accounted for the majority of the nano-codes across all four types of events, with individual cognitive factors contributing half of the nano-codes. The most common action nano-codes were confirmation bias (n=36) and failed to understand (n=36). The most common pre-condition nano-codes were channeled attention on a single issue (n=33) and inadequate communication (n=30). Conclusion Targeting quality and system improvement interventions addressing cognitive factors and team resource management as well as perceptual biases may reduce errors and further improve patient safety. These results delineate targets to further reduce never events from our healthcare system.