Testing the validity of three acute care assessment tools for assessing residents’ performance during in situ simulation: the ACAT-SimSit study

Abstract

Background The assessment of technical and nontechnical skills in emergency medicine requires reliable and usable tools. Three Acute Care Assessment Tools (ACATs) have been developed to assess medical learners in their management of cardiac arrest (ACAT-CA), coma (ACAT-coma) and acute respiratory failure (ACAT-ARF). Objective This study aims to analyze the reliability and usability of the three ACATs when used for in situ (bedside) simulation. Methods This prospective multicenter validation study tested ACATs using interprofessional in situ simulations in seven emergency departments and invited training residents to participate in them. Each session was rated by two independent raters using ACAT. Intraclass correlation coefficients (ICC) were used to assess interrater reliability, and Cronbach’s alpha coefficient was used to assess internal consistency for each ACAT. The correlation between ACATs’ scores and the learners’ level of performance was also assessed. Finally, a questionnaire and two focus groups were used to assess the usability of the ACATs. Results A total of 104 in situ simulation sessions, including 85 residents, were evaluated by 37 raters. The ICC for ACAT-CA, ACAT-coma and ACAT-ARF were 0.95 [95% confidence interval (CI), 0.93–0.98], 0.89 (95% CI, 0.77–0.95) and 0.92 (95%CI 0.83–0.96), respectively. The Cronbach’s alphas were 0.79, 0.80 and 0.73, respectively. The ACAT-CA and ARF showed good construct validity, as third-year residents obtained significantly higher scores than first-year residents (P < 0.001; P < 0.019). The raters supported the usability of the tools, even though they expressed concerns regarding the use of simulations in a summative way. Conclusion This study reported that the three ACATs showed good external validity and usability.