The Process of Developing an Assessment Checklist for Simulated Infant Respiratory Distress Using a Modified Delphi Method: A Mixed Methods Study

Abstract

Introduction Assessing clinical performance, such as managing respiratory distress, in clinical trainees is challenging yet important. Our objective was to describe and evaluate an integrative and iterative approach to developing a checklist measuring simulated clinical performance for infant respiratory distress. Methods We implemented a five-step modified Delphi process with an embedded qualitative component. An implementation period occurred followed by a second qualitative data collection. Validity evidence was collected throughout the process. Results A 19-item assessment checklist was developed for managing infant respiratory distress by medical student learners in a simulation-based setting. The iterative process provided content validity while the qualitative data provided response process validity. Cohen kappa was 0.82 indicating strong rater agreement. The assessment checklist was found to be easy to use and measure what was intended. Conclusion We developed an accurate and reliable assessment checklist for medical student learners in a simulation-based learning setting with high interrater reliability and validity evidence. Given its ease of use, we encourage medical educators and researchers to utilize this method to develop and implement assessment checklists for their interventions.

Bronchiolitis Simulation Module in the Pediatric Preclerkship Educational Exercises (PRECEDE) Curriculum

“…Interrater reliability was consistent with that reported in the checklist development process paper. 12 Overall, the checklist had a Cohen's kappa of 0.81 (95% CI, 0.75-0.88).…”

“…To date, this is the only published SBL that both reflects current guidelines 4 and includes a performance-based assessment tool with sound validity evidence. 12 Although this case and assessment tool (and its associated validity evidence) were developed specifically for pediatric clerkship students, we believe both can be easily adjusted to fit other learner groups. For example, the continued escalation of respiratory support to include high-flow nasal cannula, continuous positive pressure ventilation, or intubation can be integrated to accommodate more advanced participants.…”

“…A 19-item weighted checklist was developed via a modified Delphi approach yielding content, construct, internal consistency, and response process validity evidence ( Appendix D ). 12 Fourteen items involved timed responses. The checklist was divided into three sections: Situational Awareness/General Tasks, Initial Management, and Escalation of Care.…”

“…The checklist was easy to use, with little to no rater training needed, as shown by the strong interrater reliability and the validity evidence described in the checklist development publication. 12 A brief rater training to provide an overview and to gain familiarity with the checklist may be beneficial, especially if the simulation case is unfamiliar. Each learner group had one rater (three raters for each session).…”

Eficiencia del entrenamiento simulado del parto vaginal en estudiantes de enfermería/obstetricia

“…Mediante un panel de expertos (10 profesores expertos en atención del parto vaginal) se destacaron 9 procesos críticos esenciales en la atención segura del parto y se estableció el umbral de mínima competencia que los estudiantes debían alcanzar, usando el método Delphi (Jeffers et al, 2020) (tabla 1). El método Delphi permite a expertos lograr un consenso luego de dos o más rondas de discusión (Jeffers et al, 2020). El panel de expertos definió que el umbral de mínima competencia correspondía a lograr el 95% de los procesos de la pauta de cotejo, incluyendo todos los ítems críticos.…”