Validity and reliability of undergraduate performance assessments in an anesthesia simulator

Abstract

225Purpose: To examine the validity and reliability of performance assessment of undergraduate students using the anesthesia simulator as an evaluation tool.Methods: After ethics approval and informed consent, 135 final year medical students and 5 elective students participated in a videotaped simulator scenario with a Link-Med Patient Simulator (CAE-Link Corporation). Scenarios were based on published educational objectives of the undergraduate curriculum in anesthesia at the University of Toronto. During the simulator sessions, faculty followed a script guiding student interaction with the mannequin. Two faculty independently viewed and evaluated each videotaped performance with a 25-point criterion-based checklist. Means and standard deviations of simulatorbased marks were determined and compared with clinical and written evaluations received during the rotation. Internal consistency of the evaluation protocol was determined using inter-item and item-total correlations and correlations of specific simulator items to existing methods of evaluation.Results: Mean reliability estimates for single and average paired assessments were 0.77 and 0.86 respectively. Means of simulator scores were low and there was minimal correlation between the checklist and clinical marks (r=0.13), checklist and written marks (r=0.19) and clinical and written marks (r=0.23). Inter-item and item-total correlations varied widely and correlation between simulator items and existing evaluation tools was low.Conclusions: Simulator checklist scoring demonstrated acceptable reliability. Low correlation between different methods of evaluation may reflect reliability problems with the written and clinical marks, or that different aspects are being tested. The performance assessment demonstrated low internal consistency and further work is required. Conclusion : La cotation selon les critères liés au simulateur a démontré une fiabilité acceptable. Une faible corrélation entre différentes méthodes d'évaluation peut traduire des problèmes de fiabilité avec les notes obtenues aux examens écrits et cliniques, ou le fait que d'autres aspects sont testés. L'évaluation de la performance a démontré une faible conformité interne et la nécessité d'expérimentation ultérieure.

Simulation performance checklist generation using the Delphi technique

“…9 When considering the importance of high-stakes performance based examinations, it is critical that performance tools have adequate validity and reliability. [15][16][17] The Delphi technique assures that the identified items have high face validity and when consensus is achieved, concurrent validity is evident. 18,19 Inter-rater reliability using the newly developed checklists was high, mirroring the results of other simulation studies.…”

“…18,19 Inter-rater reliability using the newly developed checklists was high, mirroring the results of other simulation studies. 17,20 Future studies with more subjects will be conducted using a generalizability framework for analysis.…”

Psychometric characteristics of simulation‐based assessment in anaesthesia and accuracy of self‐assessed scores*

“…The cases lasted 5 min, with specified criteria for evaluation, diagnosis and treatment. Morgan and Cleave-Hogg also generated high levels of agreement between judges when assessing medical students in simulated anaesthesia problems with clear treatment pathways [15]. In contrast, Gaba et al [1] found that agreement between judges was lower when assessing anaesthetists in simulations where there was more than one possible treatment pathway and where performance varied over time.…”

“…Although previous studies have assessed aspects of simulator performance in anaesthetists [1-6, 17, 18] and medical undergraduates [9,15,19], only one has included self-evaluation. MacDonald et al [19] found that medical undergraduates' self-assessed scores of simple technical manipulations became more accurate when they had developed some expertise with the skill.…”

A worldwide survey of the use of simulation in anesthesia

“…D Di is sc cu us ss si io on n The HPS has not been demonstrated to be a superior tool for learning when compared to existing methods and further study regarding the validity of simulation performance assessments is needed. [3][4][5][6] Despite limited research involving the HPS, acquisition of this expensive technology has increased exponentially. Our search located 158 centres whereas in 1997, Shimada et al identified only 29 institutions involved in education and training of physicians in anesthesia and other specialties and allied medical professionals.…”