Simulation-based Assessment to Measure Proficiency in Mechanical Ventilation among Residents

Abstract

Background Mechanical ventilation (MV) skills are essential for clinicians caring for critically ill patients, yet few training programs use structured curricula and appropriate assessments. Objective structured clinical exams (OSCEs) have been used to assess clinical competency in many areas, but there are no OSCE models focused on MV. Objective To develop and validate a simulation-based assessment (SBA) with an OSCE structure to assess baseline MV competence among residents and identify knowledge gaps. Methods We developed an SBA using a lung simulator and a mechanical ventilator, and an OSCE structure, with six clinical scenarios in MV. We included internal medicine residents at the beginning of their rotation in the respiratory intensive care unit (ICU) of a university-affiliated hospital. A subset of residents was also evaluated with a validated multiple-choice exam (MCE) at the beginning and at the end of the ICU rotation. Scores on both assessments were normalized to range from 0 to 10. We used Cronbach’s α coefficient to assess reliability and Spearman correlation to estimate the correlation between the SBA and the MCE. Results We included 80 residents, of whom 42 also completed the MCE examinations. The final version of the SBA had 32 items, and the Cronbach’s α coefficient was 0.72 (95% confidence interval [CI], 0.64–0.81). The average SBA score was 6.2 ± 1.3, and performance was variable across items, with 80% correctly adjusting initial ventilatory settings and only 12% correctly identifying asynchrony. The MCE had 24 questions, and the average score was 7.6 ± 2.4 at the beginning of the rotation and 8.2 ± 2.3 at the end of the rotation (increase of 0.6 points; 95% CI, 0.30–0.90; P < 0.001). There was moderate correlation between the SBA and the MCE (rho = 0.41; P = 0.002). Conclusion We developed and validated an objective structured assessment on MV using a pulmonary simulator and a mechanical ventilator addressing the main competencies in MV. The performance of residents in the SBA at the beginning of an ICU rotation was lower than the performance in MCE, highlighting the need for greater emphasis on practical skills in MV during residency.

Mixed-methods education of mechanical ventilation for residents in the era of the COVID-19 pandemic: Preliminary interventional study

“…In future research, the number of participants should be increased, and educational effects of the educational program, especially of a low-fidelity simulation with little evidence. In addition, the evaluation of educational effects could be improved, e.g., by using a simulation-based assessment [ 21 ]. Furthermore, the long-term educational effects should be evaluated with an attention to their association with the rotating departments before and after the intervention.…”

Simulation-Based Teaching and Learning in Respiratory Care Education: A Narrative Review

“… 55 Skills such as measuring pulmonary mechanics, initiation, and adjustment of invasive MV settings based on case scenarios improved with simulation, while a higher percentage of errors occurred in managing patient-ventilator asynchronies, troubleshooting, and disease-specific management. 60 …”

Assessing Mechanical Ventilation Management Skills: More Tools for the Toolbox

“…In this issue of ATS Scholar , Hayashi and colleagues developed an objective structured clinical examination using a simulation-based assessment (SBA) of baseline MV management skills among residents rotating in the ICU ( 9 ). They used a modified Delphi technique to create six unique clinical cases and a 32-item checklist to assess MV management skills.…”