The promotion of competency of nurses and other health-care professionals is a goal shared by many stakeholders. In nursing, observation-based assessments are often better suited than paper-and-pencil tests for assessing many clinical abilities. Unfortunately, few instruments for simulation-based assessment of competency have been published that have undergone stringent reliability and validity evaluation. Reliability analyses typically involve some measure of rater agreement, but other sources of measurement error that affect reliability should also be considered. The purpose of this study is three-fold. First, using extant data collected from 18 nurses evaluated on 3 Scenarios by 3 Raters, we utilize generalizability (G) theory to examine the psychometric characteristics of the Nursing Performance Profile, a simulation-based instrument for assessing nursing competency. Results corroborated findings of previous studies of simulation-based assessments showing that obtaining desired score reliability requires substantially greater numbers of scenarios and/or raters. Second, we provide an illustrative exemplar of how G theory can be used to understand the relative magnitudes of sources of error variance-such as scenarios, raters, and items-and their interactions. Finally, we offer general recommendations for the design and psychometric study of simulation-based assessments in health-care contexts.

No abstract

This article advances and demonstrates a validation process to guide the development of health care simulation scenarios for assessing performance competency. The development and evaluation of each scenario used in a simulation-based competency assessment must be based on multiple sources of evidence that support the validity of the assessment for its intended use. Procedures are proposed to optimize the validity of simulation-based assessments by linking the scenario directly to the instrument and using a systematic approach for gathering and processing input from experts in the field. This validation process is then applied to the development of an original scenario for use in an assessment of nursing competency that targets objectives through patient simulation scenarios scored by multiple raters.

BackgroundDue to the difficulty of studying incentives in practice, there is limited empirical evidence of the full-impact pay-for-performance (P4P) incentive systems.ObjectiveTo evaluate the impact of P4P in a controlled, simulated environment.DesignWe employed a simulation-based randomised controlled trial with three standardised patients to assess advanced practice providers’ performance. Each patient reflected one of the following: (A) indicated for P4P screenings, (B) too young for P4P screenings, or (C) indicated for P4P screenings, but screenings are unrelated to the reason for the visit. Indication was determined by the 2016 Centers for Medicare and Medicaid Services quality measures.InterventionThe P4P group was paid $150 and received a bonus of $10 for meeting each of five outcome measures (breast cancer, colorectal cancer, pneumococcal, tobacco use and depression screenings) for each of the three cases (max $300). The control group received $200.SettingLearning resource centre.Participants35 advanced practice primary care providers (physician assistants and nurse practitioners) and 105 standardised patient encounters.MeasurementsAdherence to incentivised outcome measures, interpersonal communication skills, standards of care, and misuse.ResultsThe Type a patient was more likely to receive indicated P4P screenings in the P4P group (3.82 out of 5 P4P vs 2.94 control, p=0.02), however, received lower overall standards of care under P4P (31.88 P4P vs 37.06 control, p=0.027). The Type b patient was more likely to be prescribed screenings not indicated, but highlighted by P4P: breast cancer screening (47% P4P vs 0% control, p<0.01) and colorectal cancer screening (24% P4P vs 0% control, p=0.03). The P4P group over-reported completion of incentivised measures resulting in overpayment (average of $9.02 per patient).LimitationsA small sample size and limited variability in patient panel limit the generalisability of findings.ConclusionsOur findings caution the adoption of P4P by highlighting the unintended consequences of the incentive system.

Simulation is a critical component of nursing and medical education used to teach skills and assess student performance. In March 2020, faculty members—including the authors—at the Grace Center for Innovation in Nursing Education at the Edson College of Nursing and Health Innovation (Edson College) simulation programs at Arizona State University quickly responded to the crisis presented by COVID-19. Within a few days, all nursing simulations were transitioned from a predominately in-person design to fully online. Maintaining simulation activities throughout the first several months of the pandemic allowed students at Edson College to meet clinical assessment objectives. This transition, implemented in 2 phases, included a detailed plan of action for all Doctor of Nursing Practice (DNP) nurse practitioner objective structured clinical exams (OSCEs). The challenges required innovative planning and flexibility while maintaining the integrity of the OSCE and simulation experience. The methods implemented out of necessity are now an important part of the authors’ curricular toolbox, providing options for continued and future educational practice. This paper details the simulations designed and implemented in 2 DNP programs: the family nurse practitioner and acute care pediatric nurse practitioner programs.