Assessment of Emergency Medicine Resident Performance in a Pediatric In Situ Simulation Using Multi-Source Feedback

Abstract

IntroductionMulti-source feedback (MSF) is an evaluation method mandated by the Accreditation Council for Graduate Medical Education (ACGME). The Queen's Simulation Assessment Tool (QSAT) has been validated as being able to distinguish between resident performances in a simulation setting. The QSAT has also been demonstrated to have excellent MSF agreement when used in an adult simulation performed in a simulation lab. Using the QSAT, this study sought to determine the degree of agreement of MSF in a single pediatric (Peds) simulation case conducted in situ in a Peds emergency department (ED). MethodsThis Institutional Review Board-approved study was conducted in a four-year emergency medicine residency. A Peds resuscitation case was developed with specific behavioral anchors on the QSAT, which uses a 1-5 scale in each of five categories: Primary Assessment, Diagnostic Actions, Therapeutic Actions, Communication, and Overall Assessment. Data was gathered from six participants for each simulation. The lead resident self-evaluated and received MSF from a junior peer resident, a fixed Peds ED nurse, a random ED nurse, and two faculty (one fixed, the other from a dyad). The agreement was calculated with intraclass correlation coefficients (ICC). ResultsThe simulation was performed on 35 separate days over two academic years. A total of 106 MSF participants were enrolled. Enrollees included three faculty members, 35 team leaders, 34 peers, 33 ED registered nurses (RN), and one Peds RN; 50% of the enrollees were female (n=53). Mean QSAT scores ranged from 20.7 to 23.4. A fair agreement was demonstrated via ICC; there was no statistically significant difference between sources of MSF. Removing self-evaluation led to the highest ICC. ICC for any single or grouped non-faculty source of MSF was poor. ConclusionUsing the QSAT, the findings from this single-site cohort suggest that faculty must be included in MSF. Selfevaluation appears to be of limited value in MSF with the QSAT. The degree of MSF agreement as gathered by the QSAT was lower in this cohort than previously reported for adult simulation cases performed in the simulation lab. This may be due to either the pediatric nature of the case, the location of the simulation, or both.

The 5Cs: an assessment of residents’ ability to communicate during virtual consultations

“…In this planned secondary analysis, we evaluated resident communication performance by measuring the inter-rater reliability of the 5Cs rubric used during phone consultation and admission in two standardized simulation cases: an adult case done in the simulation lab (sim lab) and a pediatric simulation case performed in-situ in a pediatric emergency department. The simulated cases involved resuscitation as instructed in ACLS and PALS [9,10] and the primary outcome of multi-sourced feedback on performance has been previously reported [11,12]. The cases required phone consultation to EM toxicologists and a presentation for admission to attending critical care physicians.…”