Validation of a Detailed Scoring Checklist for Use During Advanced Cardiac Life Support Certification

Abstract

Introduction Defining valid, reliable, defensible, and generalizable standards for the evaluation of learner performance is a key issue in assessing both baseline competence and mastery in medical education. However, prior to setting these standards of performance, the reliability of the scores yielding from a grading tool must be assessed. Accordingly, the purpose of this study was to assess the reliability of scores generated from a set of grading checklists used by non-expert raters during simulations of American Heart Association (AHA) MegaCodes. Methods The reliability of scores generated from a detailed set of checklists, when used by four non-expert raters, was tested by grading team leader performance in eight MegaCode scenarios. Videos of the scenarios were reviewed and rated by trained faculty facilitators and by a group of non-expert raters. The videos were reviewed “continuously” and “with pauses.” Two content experts served as the reference standard for grading, and four non-expert raters were used to test the reliability of the checklists. Results Our results demonstrate that non-expert raters are able to produce reliable grades when using the checklists under consideration, demonstrating excellent intra-rater reliability and agreement with a reference standard. The results also demonstrate that non-expert raters can be trained in the proper use of the checklist in a short amount of time, with no discernible learning curve thereafter. Finally, our results show that a single trained rater can achieve reliable scores of team leader performance during AHA MegaCodes when using our checklist in continuous mode, as measures of agreement in total scoring were very strong (Lin’s Concordance Correlation Coefficient = 0.96; Intraclass Correlation Coefficient = 0.97). Discussion We have shown that our checklists can yield reliable scores, are appropriate for use by non-expert raters, and are able to be employed during continuous assessment of team leader performance during the review of a simulated MegaCode. This checklist may be more appropriate for use by Advanced Cardiac Life Support (ACLS) instructors during MegaCode assessments than current tools provided by the AHA.

Assessment Tools for Use During Anesthesia-Centric Pediatric Advanced Life Support Training and Evaluation

“…We based the methods for developing the assessment checklists on the methodology of similar studies and summarize our methods here. 4,9,10 We established the content of the checklists through a detailed review of relevant literature to identify evidence-based practices and further refined them through a modified Delphi process. 2,9–14 We used the checklists to score several pilot scenarios (described below) and made iterative changes based on observations from these pilot simulations.…”

“…They received training in the manner previously described. 9 We used scores from the training videos to gauge the effectiveness of the rater training. The raters spent approximately 4 hours in a training period in which they viewed and scored 4 training videos and resolved all discrepancies between raters.…”

“…1,2 The rarity of these events makes it difficult, if not impossible, for clinicians to develop the skills required for optimal management and for training programs to assess provider performance during actual resuscitation events, 3,4 leading some to propose resuscitation guidelines and curricula tailored to the perioperative environment and suggesting that clinicians receive more frequent resuscitation training with skills assessment. 2–6 The literature contains reports of several scoring instruments for assessing the resuscitation performance of providers in managing both adult and pediatric patients in a variety of settings 4,7–9 ; however, there are no valid published tools for the assessment of pediatric resuscitation skills in the perioperative setting with demonstrated reliability. Accordingly, we describe here our methods for developing a set of assessment tools for measuring participant competency during pediatric perioperative resuscitation scenarios and the process of assessing the reliability of scores produced by these tools.…”

A Tool to Assess Student Performance in a Clostridium difficile Infection Simulation Scenario

“…21,22 Also, reliable scores can be obtained through the use of an ACLS checklist during a high stakes performance assessment, called a Megacode, used during ACLS certification. 23 To this end, the need for valid, reliable assessment tools to evaluate skills-based performance of student pharmacists during simulation cannot be overstated, particularly as pharmacy education programs are called to provide evidence of competency/mastery of pre-APPE domains. Future studies based on these results are needed to target the best approach for using evaluation tools to set standards of proficiency in pharmacy practice.…”

Internet Training Resulted in Improved Trainee Performance in a Simulated Opioid-Poisoned Patient as Measured by Checklist

“…Moreover, medical simulation poses no risk of harm to patients. Simulation has been used as an assessment tool to measure the performance of medical trainees in pediatric resuscitation, perioperative emergencies, and adult medical resuscitations [8][9][10].…”