Abstract:SummaryAssessment tools must be investigated for reliability, validity and feasibility before being implemented. In 2013, the Australian and New Zealand College of Anaesthetists introduced workplace-based assessments, including a direct observation of a procedural skills assessment tool. The objective of this study was to evaluate the psychometric properties of this assessment tool for ultrasound-guided regional anaesthesia. Six experts assessed 30 video-recorded trainee performances of ultrasound-guided regio… Show more
“…Students were instructed to suture at least one complete knot under the microscope which was assessed on an individual basis by a consultant neurosurgeon. Performance was assessed using the Direct Observation of Procedural Skills (DOPS) tool [2, 13, 23, 25, 31], which is part of the Intercollegiate Surgical Curriculum Program (ISCP). This is a validated tool which forms a core aspect of postgraduate workplace-based assessment in the UK.Fig.…”
IntroductionPhysiological hand tremor occurs naturally, due to oscillations of the upper extremities. Tremor can be exacerbated by stress and anxiety, interfering with fine motor tasks and potentially impact on surgical performance, particularly in microsurgery. We investigated the link between tremor, anxiety and performance in a neurosurgical module as part of an international surgical course.MethodsEssential Skills in the Management of Surgical Cases (ESMSC) course recruits medical students from European Union (EU) medical schools. Students are asked to suture the dura mater in an ex vivo swine model, of which the first suture completed was assessed. Questionnaires were distributed before and after the module, eliciting tremor risk factors, self-perception of tremor and anxiety. Johnson O’Connor dexterity pad was used to objectively measure dexterity. Direct Observation of Procedural Skills (DOPS) was used to assess skills-based performance. Anxiety was assessed using the Westside Test Anxiety Scale (WTAS). Tremor was evaluated by four qualified neurosurgeons.ResultsForty delegates participated in the study. Overall performance decreased with greater subjective perception of anxiety (p = 0.032, rho = − 0.392). Although increasing scores for tremor at rest and overall WTAS score were associated with decreased performance, this was not statistically significant (p > 0.05). Tremor at rest did not affect dexterity (p = 0.876, rho = − 0.027).ConclusionsPhysiological tremor did not affect student performance and microsurgical dexterity in a simulation-based environment. Self-perception of anxiety affected performance in this module, suggesting that more confident students perform better in a simulated neurosurgical setting.
“…Students were instructed to suture at least one complete knot under the microscope which was assessed on an individual basis by a consultant neurosurgeon. Performance was assessed using the Direct Observation of Procedural Skills (DOPS) tool [2, 13, 23, 25, 31], which is part of the Intercollegiate Surgical Curriculum Program (ISCP). This is a validated tool which forms a core aspect of postgraduate workplace-based assessment in the UK.Fig.…”
IntroductionPhysiological hand tremor occurs naturally, due to oscillations of the upper extremities. Tremor can be exacerbated by stress and anxiety, interfering with fine motor tasks and potentially impact on surgical performance, particularly in microsurgery. We investigated the link between tremor, anxiety and performance in a neurosurgical module as part of an international surgical course.MethodsEssential Skills in the Management of Surgical Cases (ESMSC) course recruits medical students from European Union (EU) medical schools. Students are asked to suture the dura mater in an ex vivo swine model, of which the first suture completed was assessed. Questionnaires were distributed before and after the module, eliciting tremor risk factors, self-perception of tremor and anxiety. Johnson O’Connor dexterity pad was used to objectively measure dexterity. Direct Observation of Procedural Skills (DOPS) was used to assess skills-based performance. Anxiety was assessed using the Westside Test Anxiety Scale (WTAS). Tremor was evaluated by four qualified neurosurgeons.ResultsForty delegates participated in the study. Overall performance decreased with greater subjective perception of anxiety (p = 0.032, rho = − 0.392). Although increasing scores for tremor at rest and overall WTAS score were associated with decreased performance, this was not statistically significant (p > 0.05). Tremor at rest did not affect dexterity (p = 0.876, rho = − 0.027).ConclusionsPhysiological tremor did not affect student performance and microsurgical dexterity in a simulation-based environment. Self-perception of anxiety affected performance in this module, suggesting that more confident students perform better in a simulated neurosurgical setting.
“…These video recordings were also used to evaluate an assessment tool not specific for regional anesthesia. 9 …”
Section: Methodsmentioning
confidence: 99%
“…There was evidence to support construct validity. Domains 7,9,17,20, and 22 were deconstructed and expanded into components that were assessed individually. The descriptors for each item were worded to prompt the assessor to take note of QCBs.…”
Background and ObjectivesChecklists and global rating scales (GRSs) are used for assessment of anesthesia procedural skills. The purpose of this study was to evaluate the reliability and validity of a recently proposed assessment tool comprising a checklist and GRS specific for ultrasound-guided regional anesthesia.MethodsIn this prospective, fully crossed study, we videotaped 30 single-target nerve block procedures performed by anesthesia trainees. Following pilot assessment and observer training, videos were assessed in random order by 6 blinded, expert observers. Interrater reliability was evaluated with intraclass correlation coefficients (ICCs) based on a 2-way random-effects model that took into account both agreement and correlation between observer results. Construct validity and feasibility were also evaluated.ResultsThe ICC between assessors’ total scores was 0.44 (95% confidence interval, 0.27–0.62). All 6 observers scored “experienced trainees” higher than “inexperienced trainees” (median total score 76.7 vs 54.2, P = 0.01), supporting the test’s construct validity. The median time to assess the videos was 4 minutes 29 seconds.ConclusionsThis is the first study to evaluate the reliability and validity of a combined checklist and GRS for ultrasound-guided regional anesthesia using multiple observers and taking into account both absolute agreement and correlation in determining the ICC of 0.44 for interrater reliability. There was evidence to support construct validity.
“…Reliability of a tool is its ability to assess skills consistently by different assessors at different times while construct validity is the ability of the tool to differentiate among varying levels of expertise. [67111213]…”
Background and Aims:Gaining expertise in procedural skills is essential for achieving clinical competence during anesthesia training. Supervisors have the important responsibility of deciding when the trainee can be allowed to perform various procedures without direct supervision while ensuring patient safety. This requires robust and reliable assessment techniques. Airway management with bag-mask ventilation and tracheal intubation are routinely performed by anesthesia trainees at induction of anesthesia and to save lives during a cardiorespiratory arrest. The purpose of this study was to evaluate the construct validity, and inter-rater and test-retest reliability of a tool designed to assess competence in bag-mask ventilation followed by tracheal intubation in anesthesia trainees.Material and Methods:Informed consent was obtained from all participants. Tracheal intubation and bag-mask ventilation skills in 10 junior and 10 senior anesthesia trainees were assessed by two investigators on two occasions at a 3-4 weeks interval, using a procedure-specific assessment tool.Results:Average kappa value for inter-rater reliability was 0.91 and 0.99 for the first and second assessments, respectively, with an average agreement of 95%. The average agreement for test-retest reliability was 82% with a kappa value of 0.39. Senior trainees obtained higher scores compared to junior trainees in all areas of assessment, with a significant difference for patient positioning, preoxygenation, and laryngoscopy technique, depicting good construct validity.Conclusion:The tool designed to assess bag-mask ventilation and tracheal intubation skills in anesthesia trainees demonstrated excellent inter-rater reliability, fair test-retest reliability, and good construct validity. The authors recommend its use for formative and summative assessment of junior anesthesia trainees.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
