Simulation performance checklist generation using the Delphi technique

Abstract

Purpose: Performance assessment using high fidelity simulation is problematic, due to the difficulty in developing valid and reliable evaluation tools. The Delphi technique is a consensus based content generation method used for multiple purposes such as policy development, best-evidence practice guidelines and competency assessments. The purpose of this study was to develop checklists using a modified Delphi technique to evaluate the performance of practicing anesthesiologists managing two simulated scenarios. Methods:The templates for two simulation scenarios were emailed to five anesthesiologists who were asked to generate performance items. Data were collated anonymously and returned. An a priori decision was made to delete items endorsed by ≤ 20% of participants. This process of collection, collation and re-evaluation was repeated until consensus was reached. Four independent raters used the checklist to assess three subjects managing the two simulation scenarios. Interrater reliability was assessed using average measures intraclass correlation (ICC) and repeated measures analysis of variance (ANOVA) was used to assess differences in difficulty between scenarios. Results:The final checklists included 131 items for scenario 1 and 126 items for scenario 2. The mean inter-rater reliability was 0.921 for scenario 1 and 0.903 for scenario 2. Repeated measures ANOVA revealed no statistically significant difference in difficulty between scenarios.Discussion: The Delphi technique can be very useful to generate consensus based evaluation tools with high content and face validity compared to subjective evaluative tools. Since there was no difference in scenario difficulty, these scenarios can be used to determine the effect of educational interventions on performance.

The Creation of an Objective Assessment Tool for Ultrasound-Guided Regional Anesthesia Using the Delphi Method

“…9 This method was designed to encourage debate among members while keeping their identities anonymous. This way, the impact of strong opinions and personalities are eliminated, and all opinions are given equal weight.…”

Development and Validation of an Assessment Tool for Competency in Critical Care Ultrasound

“…17,18 The Delphi method is a consensus-based content generation method frequently used to develop guidelines, policy statements, and assessment tools. The modified Delphi method differs in that an initial structure is given to participants, from which feedback is generated and incorporated until consensus is reached.…”

“…This method has been used to develop performance checklists with high content-related validity evidence. 17,18 One author (P.P. ), a critical care physician and ultrasound educator, created comprehensive step-by-step procedure checklists for CCUS modules based on existing literature and published guidelines.…”

Automated Case Cancellation Review System Improves Systems-Based Practice

“…This survey was distributed to all residents completing the rotation, so no sampling method was used. The survey was created through a modified Delphi technique [20] and piloted with residents and faculty for appropriate validation [21] prior to administration. Surveys were administered using REDCap data capture tools hosted at our institution [22].…”